[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484750913 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003042] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007861] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.012842] *** VALIDATE selinux *** [ 0.020478] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027126] *** VALIDATE tmpfs *** [ 0.029404] *** VALIDATE proc *** [ 0.030256] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.032289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035029] Spectre V2 : User space: Vulnerable [ 0.036009] Speculative Store Bypass: Vulnerable [ 0.039555] debug: unmapping init [mem 0xffffffffb1059000-0xffffffffb1060fff] [ 0.041983] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042622] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.043904] ... bit width: 48 [ 0.044023] ... generic registers: 4 [ 0.044793] ... value mask: 0000ffffffffffff [ 0.045017] ... max period: 00007fffffffffff [ 0.046015] ... fixed-purpose events: 3 [ 0.046999] ... event mask: 000000070000000f [ 0.047297] rcu: Hierarchical SRCU implementation. [ 0.049442] smp: Bringing up secondary CPUs ... [ 0.050539] x86: Booting SMP configuration: [ 0.051026] .... node #0, CPUs: #1 #2 #3 [ 0.054212] smp: Brought up 1 node, 4 CPUs [ 0.056011] smpboot: Max logical packages: 1 [ 0.057020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134579] node 0 deferred pages initialised in 75ms [ 0.137071] devtmpfs: initialized [ 0.137700] x86/mm: Memory block size: 128MB [ 0.140335] gcov: version magic: 0x41383552 [ 0.142479] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146263] pinctrl core: initialized pinctrl subsystem [ 0.147172] [ 0.147746] ************************************************************* [ 0.149011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151007] ** ** [ 0.152007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154008] ** ** [ 0.155006] ** This means that this kernel is built to expose internal ** [ 0.156008] ** IOMMU data structures, which may compromise security on ** [ 0.158009] ** your system. ** [ 0.160008] ** ** [ 0.161007] ** If you see this message and you are not debugging the ** [ 0.162006] ** kernel, report this immediately to your vendor! ** [ 0.164008] ** ** [ 0.165006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166005] ************************************************************* [ 0.168613] NET: Registered protocol family 16 [ 0.169274] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177112] cpuidle: using governor menu [ 0.178307] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179388] PCI: Using configuration type 1 for base access [ 0.182197] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193233] cryptd: max_cpu_qlen set to 1000 [ 0.194291] ACPI: Added _OSI(Module Device) [ 0.195027] ACPI: Added _OSI(Processor Device) [ 0.195916] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206017] ACPI: Interpreter enabled [ 0.207050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208010] ACPI: Using IOAPIC for interrupt routing [ 0.210114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212512] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223048] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236379] acpiphp: Slot [2] registered [ 0.237104] acpiphp: Slot [5] registered [ 0.238114] acpiphp: Slot [6] registered [ 0.240118] acpiphp: Slot [7] registered [ 0.241102] acpiphp: Slot [8] registered [ 0.242087] acpiphp: Slot [9] registered [ 0.244093] acpiphp: Slot [10] registered [ 0.245126] acpiphp: Slot [3] registered [ 0.246078] acpiphp: Slot [4] registered [ 0.248098] acpiphp: Slot [11] registered [ 0.249100] acpiphp: Slot [12] registered [ 0.250101] acpiphp: Slot [13] registered [ 0.252109] acpiphp: Slot [14] registered [ 0.253103] acpiphp: Slot [15] registered [ 0.254087] acpiphp: Slot [16] registered [ 0.256090] acpiphp: Slot [17] registered [ 0.257082] acpiphp: Slot [18] registered [ 0.258056] acpiphp: Slot [19] registered [ 0.260083] acpiphp: Slot [20] registered [ 0.261089] acpiphp: Slot [21] registered [ 0.262077] acpiphp: Slot [22] registered [ 0.263074] acpiphp: Slot [23] registered [ 0.264101] acpiphp: Slot [24] registered [ 0.264890] acpiphp: Slot [25] registered [ 0.266059] acpiphp: Slot [26] registered [ 0.266859] acpiphp: Slot [27] registered [ 0.267090] acpiphp: Slot [28] registered [ 0.269062] acpiphp: Slot [29] registered [ 0.269922] acpiphp: Slot [30] registered [ 0.270063] acpiphp: Slot [31] registered [ 0.270827] PCI host bridge to bus 0000:00 [ 0.272013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.280022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.285229] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.287571] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.291220] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.300788] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.306011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.317018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320125] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322658] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327809] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.350015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355921] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421199] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467516] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.486018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.504020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.513527] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.528018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.564442] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.571016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.577019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.592027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.601410] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.612017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.640017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.650376] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.653366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.656362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.658343] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.661228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.667026] iommu: Default domain type: Passthrough [ 0.668408] SCSI subsystem initialized [ 0.670122] ACPI: bus type USB registered [ 0.671110] usbcore: registered new interface driver usbfs [ 0.673151] usbcore: registered new interface driver hub [ 0.676077] usbcore: registered new device driver usb [ 0.677176] pps_core: LinuxPPS API ver. 1 registered [ 0.679007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.681045] PTP clock support registered [ 0.682124] EDAC MC: Ver: 3.0.0 [ 0.683749] PCI: Using ACPI for IRQ routing [ 0.684855] NetLabel: Initializing [ 0.685010] NetLabel: domain hash size = 128 [ 0.686010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687143] NetLabel: unlabeled traffic allowed by default [ 0.689291] vgaarb: loaded [ 0.690284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.692014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.703126] clocksource: Switched to clocksource kvm-clock [ 0.811420] VFS: Disk quotas dquot_6.6.0 [ 0.813065] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814769] *** VALIDATE ramfs *** [ 0.816412] *** VALIDATE hugetlbfs *** [ 0.817846] pnp: PnP ACPI init [ 0.820482] pnp: PnP ACPI: found 6 devices [ 0.841857] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.845413] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.847685] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.849848] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.852311] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.854657] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.857393] NET: Registered protocol family 2 [ 0.859751] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.864814] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.868279] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.873789] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.876705] TCP: Hash tables configured (established 65536 bind 65536) [ 0.879391] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.882296] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884688] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887438] NET: Registered protocol family 1 [ 0.891259] RPC: Registered named UNIX socket transport module. [ 0.893167] RPC: Registered udp transport module. [ 0.894374] RPC: Registered tcp transport module. [ 0.895403] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.896868] NET: Registered protocol family 44 [ 0.897793] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.899109] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.900270] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.901689] PCI: CLS 0 bytes, default 64 [ 0.903282] Unpacking initramfs... [ 2.246832] debug: unmapping init [mem 0xffff8a93fcc54000-0xffff8a93fffbffff] [ 2.250428] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.252303] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.254871] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.720464] Initialise system trusted keyrings [ 2.722197] Key type blacklist registered [ 2.724278] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.733372] zbud: loaded [ 2.736364] *** VALIDATE nfs *** [ 2.737615] *** VALIDATE nfs4 *** [ 2.738991] pstore: using deflate compression [ 2.742459] Platform Keyring initialized [ 2.836352] NET: Registered protocol family 38 [ 2.837629] Key type asymmetric registered [ 2.838436] Asymmetric key parser 'x509' registered [ 2.839771] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.841987] io scheduler mq-deadline registered [ 2.842886] io scheduler kyber registered [ 2.844183] io scheduler bfq registered [ 2.845193] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.847336] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.849563] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.851454] ACPI: Power Button [PWRF] [ 2.855084] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.860508] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.874316] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.883706] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.902335] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.931476] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.960188] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.965227] Non-volatile memory driver v1.3 [ 2.966892] Linux agpgart interface v0.103 [ 2.997851] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.000402] vda: detected capacity change from 0 to 74838016 [ 3.015394] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.018158] vdb: detected capacity change from 0 to 1073741824 [ 3.031486] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.033405] vdc: detected capacity change from 0 to 2621440000 [ 3.046663] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.049293] vdd: detected capacity change from 0 to 2621440000 [ 3.063876] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.067865] vde: detected capacity change from 0 to 4294967296 [ 3.089698] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.092541] vdf: detected capacity change from 0 to 4294967296 [ 3.100381] libphy: Fixed MDIO Bus: probed [ 3.117212] usbcore: registered new interface driver usbserial_generic [ 3.119124] usbserial: USB Serial support registered for generic [ 3.121467] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.126375] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.128514] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.131104] mousedev: PS/2 mouse device common for all mice [ 3.134505] rtc_cmos 00:05: RTC can wake from S4 [ 3.136519] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.136610] rtc_cmos 00:05: registered as rtc0 [ 3.141311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.141368] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.144246] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.146674] intel_pstate: CPU model not supported [ 3.152529] hid: raw HID events driver (C) Jiri Kosina [ 3.154896] usbcore: registered new interface driver usbhid [ 3.157211] usbhid: USB HID core driver [ 3.159000] drop_monitor: Initializing network drop monitor service [ 3.161863] Initializing XFRM netlink socket [ 3.164131] NET: Registered protocol family 10 [ 3.167263] Segment Routing with IPv6 [ 3.168597] NET: Registered protocol family 17 [ 3.170419] mpls_gso: MPLS GSO support [ 3.176547] RAS: Correctable Errors collector initialized. [ 3.178665] AVX version of gcm_enc/dec engaged. [ 3.180027] AES CTR mode by8 optimization enabled [ 3.251863] sched_clock: Marking stable (3251841497, 0)->(4140200180, -888358683) [ 3.254583] registered taskstats version 1 [ 3.255848] Loading compiled-in X.509 certificates [ 3.256924] zswap: loaded using pool lzo/zbud [ 3.277858] Key type big_key registered [ 3.288073] Key type encrypted registered [ 3.289808] ima: No TPM chip found, activating TPM-bypass! [ 3.291946] ima: Allocated hash algorithm: sha1 [ 3.293506] ima: No architecture policies found [ 3.295077] evm: Initialising EVM extended attributes: [ 3.297218] evm: security.selinux [ 3.298464] evm: security.ima [ 3.299385] evm: security.capability [ 3.300643] evm: HMAC attrs: 0x1 [ 3.302834] rtc_cmos 00:05: setting system clock to 2026-09-15 13:01:55 UTC (1789477315) [ 3.308708] debug: unmapping init [mem 0xffffffffb2003000-0xffffffffb21fffff] [ 3.311613] debug: unmapping init [mem 0xffffffffb0d82000-0xffffffffb1058fff] [ 3.322079] Write protecting the kernel read-only data: 28672k [ 3.325313] debug: unmapping init [mem 0xffffffffaf403000-0xffffffffaf5fffff] [ 3.327840] debug: unmapping init [mem 0xffffffffafd14000-0xffffffffafdfffff] [ 3.363806] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.371565] systemd[1]: Detected virtualization kvm. [ 3.373267] systemd[1]: Detected architecture x86-64. [ 3.375237] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.400489] systemd[1]: No hostname configured. [ 3.401526] systemd[1]: Set hostname to . [ 3.402664] random: systemd: uninitialized urandom read (16 bytes read) [ 3.404863] systemd[1]: Initializing machine ID from random generator. [ 3.439471] random: ln: uninitialized urandom read (6 bytes read) [ 3.523215] random: systemd: uninitialized urandom read (16 bytes read) [ 3.525894] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.530474] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.534759] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.091417] device-mapper: uevent: version 1.0.3 [ 4.093322] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.734958] virtio_net virtio0 ens2: renamed from eth0 [ 4.759510] random: fast init done [ 4.793815] scsi host0: ata_piix [ 4.804576] scsi host1: ata_piix [ 4.805744] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.807644] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.471433] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.804822] random: crng init done [ 9.806296] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.079920] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.089450] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.307789] SELinux: Disabled at runtime. [ 11.363774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.368424] systemd[1]: Detected virtualization kvm. [ 11.369643] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.888683] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.892253] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.896884] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.902318] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.905565] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.912308] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.917068] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ 12.022625] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.324715] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.568945] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.642338] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.751516] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.759795] EDAC sbridge: Ver: 1.1.2 [ 14.118304] Key type dns_resolver registered [ 14.421621] NFS: Registering the id_resolver key type [ 14.423559] Key type id_resolver registered [ 14.424879] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 39.006463] libcfs: loading out-of-tree module taints kernel. [ 39.025479] Key type ._llcrypt registered [ 39.026859] Key type .llcrypt registered [ 39.073444] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 47.812582] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 48.458909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.466927] alg: No test for adler32 (adler32-zlib) [ 49.491260] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 49.830282] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 51.455182] Key type lgssc registered [ 52.247651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.560585] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.533321] hrtimer: interrupt took 5451309 ns [ 117.976538] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 130.653539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 130.713351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.001398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.049583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.126770] Lustre: lustre-MDT0000: new disk, initializing [ 132.178772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.191819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.463603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.851059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.940453] Lustre: 6491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.967347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.970957] Lustre: Skipped 1 previous similar message [ 149.033664] Lustre: lustre-MDT0001: new disk, initializing [ 149.081292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.110220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.122183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.129509] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.597458] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.678041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.088532] Lustre: lustre-OST0000: new disk, initializing [ 168.093612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.105514] Lustre: 8431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.185215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.573775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.586932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.644989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.885708] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.157548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.369222] Lustre: lustre-OST0001: new disk, initializing [ 191.380027] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.386243] Lustre: 9504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.484830] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.290878] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.740465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.746905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 200.803034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.461427] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.817290] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 226.411350] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.121546] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 236.651195] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 238.977689] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 241.298866] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 243.261861] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 09:05:54 EDT 2026 [ 261.476873] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 263.044265] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 282.080584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.090533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.137624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.613525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.641592] Lustre: Skipped 2 previous similar messages [ 282.644840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.648973] Lustre: Skipped 2 previous similar messages [ 284.654966] Lustre: server umount lustre-MDT0000 complete [ 288.394859] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477600 with bad export cookie 12960279158112887997 [ 288.399382] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 288.412824] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 292.832720] LustreError: 6499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.835878] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.837432] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 292.865524] LustreError: 6499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 292.928638] Lustre: Skipped 1 previous similar message [ 293.857628] LustreError: 8106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.883634] LustreError: 8106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 294.839728] Lustre: server umount lustre-MDT0001 complete [ 298.721654] Lustre: server umount lustre-OST0000 complete [ 302.207795] Lustre: server umount lustre-OST0001 complete [ 309.014388] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 318.519740] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 355.919145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.184731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 356.220320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 356.348661] Lustre: lustre-MDT0000: new disk, initializing [ 356.443748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.465543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.188798] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 371.805640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.972264] Lustre: 15056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 372.044138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 372.049335] Lustre: Skipped 1 previous similar message [ 372.203445] Lustre: lustre-MDT0001: new disk, initializing [ 372.328512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 372.388079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 372.407986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 376.920768] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 386.753249] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 392.735748] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 399.800393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 400.114898] Lustre: lustre-OST0000: new disk, initializing [ 400.119924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 400.128311] Lustre: 17024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 400.242457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.964517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 401.981572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 402.161543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 406.927714] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 416.528784] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 421.099567] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 421.294818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 425.750279] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 425.927720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 428.000977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.004377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.016485] LustreError: Skipped 1 previous similar message [ 428.016883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 428.016887] Lustre: Skipped 3 previous similar messages [ 428.047434] Lustre: Skipped 1 previous similar message [ 433.123745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.138070] Lustre: Skipped 2 previous similar messages [ 433.800330] Lustre: server umount lustre-OST0000 complete [ 438.246779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.264021] Lustre: Skipped 1 previous similar message [ 443.363705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 443.375098] Lustre: Skipped 2 previous similar messages [ 443.389521] LustreError: 15064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.584060] Lustre: server umount lustre-MDT0000 complete [ 446.945570] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477759 with bad export cookie 12960279158112889530 [ 446.955812] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.964874] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 447.292037] Lustre: server umount lustre-MDT0001 complete [ 450.958522] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:09:22 (1789477762) [ 453.773339] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 455.867476] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:09:27 (1789477767) [ 462.342613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.796822] LustreError: 19434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.867313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.466292] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 467.947534] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.997867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.345717] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 486.290568] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.789628] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 498.558697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.108270] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 511.072651] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 518.625696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.628582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.660435] Lustre: Skipped 1 previous similar message [ 524.913303] Lustre: server umount lustre-OST0000 complete [ 534.706312] Lustre: server umount lustre-MDT0000 complete [ 537.806169] LustreError: 19416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477850 with bad export cookie 12960279158112890405 [ 537.812432] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.817337] LustreError: 19416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 538.186189] Lustre: server umount lustre-MDT0001 complete [ 548.538933] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 551.301549] Key type lgssc unregistered [ 551.568580] LNet: 22971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.576407] LNetError: 22971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.588222] LNet: Removed LNI 192.168.201.118@tcp [ 552.687279] Key type .llcrypt unregistered [ 552.688535] Key type ._llcrypt unregistered [ 573.666703] Key type ._llcrypt registered [ 573.669703] Key type .llcrypt registered [ 573.764324] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 574.745423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 574.894124] alg: No test for adler32 (adler32-zlib) [ 575.948321] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 576.165100] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 577.848114] Key type lgssc registered [ 579.103691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 587.783569] loop: module loaded [ 587.889301] loop0: detected capacity change from 0 to 204800000 [ 588.143312] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.160435] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.385410] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 594.456858] loop0: detected capacity change from 0 to 204800000 [ 594.666234] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 594.688740] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 594.697872] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.141545] loop0: detected capacity change from 0 to 204800000 [ 606.669710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 606.690053] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.079431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 608.187391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 608.256952] Lustre: lustre-MDT0000: new disk, initializing [ 608.381240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.426319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 613.777372] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 622.639461] loop1: detected capacity change from 0 to 204800000 [ 627.647386] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 628.008588] Lustre: lustre-OST07c6: new disk, initializing [ 628.012905] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 628.018696] Lustre: Skipped 1 previous similar message [ 628.024338] Lustre: 26154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 628.110450] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 634.318188] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 635.494520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 635.521946] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 635.562925] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 639.426359] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 639.435500] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 639.444655] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 639.456098] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 639.467306] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 639.477066] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 655.737787] Lustre: 25141:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 655.748521] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 655.758454] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 655.765243] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 655.770563] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 655.776186] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 660.312334] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 660.330873] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 660.339915] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 660.348321] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.359908] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 660.372600] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.386584] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 660.395408] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.418042] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 660.428214] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.442897] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 660.451120] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.683244] Lustre: 25141:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 663.691591] Lustre: 25141:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 663.695739] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 663.707538] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.719618] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 663.724168] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.728584] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 663.733812] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.745445] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 663.756888] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.761664] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 663.767543] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.478198] Lustre: 26162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 668.494310] Lustre: 26162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 668.499966] Lustre: 26162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 668.509334] Lustre: 26162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.520429] Lustre: 26162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 668.527695] Lustre: 26162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.542524] Lustre: 26162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 668.557459] Lustre: 26162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.570702] Lustre: 26162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 668.579148] Lustre: 26162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.590752] Lustre: 26162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.599579] Lustre: 26162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.643292] Lustre: 26162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 679.667279] Lustre: 26162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 679.674842] Lustre: 26162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 679.686115] Lustre: 26162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.696073] Lustre: 26162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.712949] Lustre: 26162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.726460] Lustre: 26162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 679.739590] Lustre: 26162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.746658] Lustre: 26162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 679.754630] Lustre: 26162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.765619] Lustre: 26162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.773640] Lustre: 26162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 727.530164] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 727.538724] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.561801] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 732.773459] Lustre: server umount lustre-OST07c6 complete [ 742.666063] Lustre: server umount lustre-MDT0000 complete [ 759.419677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 762.487866] Key type lgssc unregistered [ 762.735263] LNet: 28719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.745323] LNetError: 28719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.762437] LNet: Removed LNI 192.168.201.118@tcp [ 763.630424] Key type .llcrypt unregistered [ 763.633741] Key type ._llcrypt unregistered [ 781.385978] Key type ._llcrypt registered [ 781.388102] Key type .llcrypt registered [ 781.550456] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 782.959985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.003374] alg: No test for adler32 (adler32-zlib) [ 784.107603] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 784.358939] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 786.063160] Key type lgssc registered [ 787.161721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 794.394391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 794.425863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.831552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.099939] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 805.346312] LustreError: 29984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.363438] LustreError: 29984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 808.091452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.574779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 813.624699] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 819.763427] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 825.750765] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 832.320929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 832.833447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.002813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 840.547144] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 848.266543] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 864.738131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 864.748297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.761981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 865.763283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.768854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.769525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.970807] Lustre: server umount lustre-OST0000 complete [ 874.986441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 874.996287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.009337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 875.020854] Lustre: Skipped 1 previous similar message [ 877.717376] Lustre: server umount lustre-MDT0000 complete [ 881.088574] LustreError: 29963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478193 with bad export cookie 14372228129854369088 [ 881.090566] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 881.096534] LustreError: 29963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 881.454606] Lustre: server umount lustre-MDT0001 complete [ 892.204698] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 894.629523] Key type lgssc unregistered [ 894.811936] LNet: 33721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.821403] LNetError: 33721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.838714] LNet: Removed LNI 192.168.201.118@tcp [ 895.681825] Key type .llcrypt unregistered [ 895.684154] Key type ._llcrypt unregistered [ 911.643718] Key type ._llcrypt registered [ 911.649155] Key type .llcrypt registered [ 911.741553] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 912.775603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.879633] alg: No test for adler32 (adler32-zlib) [ 913.966331] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 914.177409] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 915.887228] Key type lgssc registered [ 917.287558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.648163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.675253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.238671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.314775] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 937.449934] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.481922] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 940.883879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.293753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 946.059793] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 955.122983] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 961.336162] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 968.309026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 969.098578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.170137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 977.188479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 986.487759] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 996.834803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.848297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 996.850407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.955493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.958228] Lustre: Skipped 2 previous similar messages [ 1002.895576] Lustre: server umount lustre-OST0000 complete [ 1007.073750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.079179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.084761] Lustre: Skipped 2 previous similar messages [ 1012.196158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.200180] Lustre: Skipped 1 previous similar message [ 1012.814630] Lustre: server umount lustre-MDT0000 complete [ 1016.717614] LustreError: 34954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478328 with bad export cookie 4960068333403473462 [ 1016.720891] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.724430] LustreError: 34954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1016.976170] Lustre: server umount lustre-MDT0001 complete [ 1029.345984] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1032.973314] Key type lgssc unregistered [ 1033.411983] LNet: 38512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.421899] LNetError: 38512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.446592] LNet: Removed LNI 192.168.201.118@tcp [ 1034.595374] Key type .llcrypt unregistered [ 1034.597547] Key type ._llcrypt unregistered [ 1061.104937] Key type ._llcrypt registered [ 1061.107726] Key type .llcrypt registered [ 1061.285721] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1062.691132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.754990] alg: No test for adler32 (adler32-zlib) [ 1063.843434] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1064.155370] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1065.946235] Key type lgssc registered [ 1067.240496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.924120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1074.945667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.410214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.099672] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1085.921743] LustreError: 39930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.931901] LustreError: 39930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1088.995207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.380104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1093.097562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1101.966780] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1109.516610] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1116.159134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1116.607380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.720542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1124.122923] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1133.462302] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1139.148885] Lustre: server umount lustre-MDT0000 complete [ 1139.169215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.172152] LustreError: 39925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.174678] Lustre: Skipped 2 previous similar messages [ 1139.180012] LustreError: 39925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1140.325282] LustreError: 39925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.621494] LustreError: 39912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478454 with bad export cookie 17186463842111139557 [ 1142.643964] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.291900] LustreError: 41363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.292360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.335370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1145.447348] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 1147.302386] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1148.898609] Lustre: server umount lustre-MDT0001 complete [ 1170.140698] Lustre: server umount lustre-OST0000 complete [ 1191.165883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1194.380232] Key type lgssc unregistered [ 1194.875703] LNet: 43794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.897587] LNetError: 43794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.939392] LNet: Removed LNI 192.168.201.118@tcp [ 1196.098113] Key type .llcrypt unregistered [ 1196.100385] Key type ._llcrypt unregistered [ 1220.153480] Key type ._llcrypt registered [ 1220.155455] Key type .llcrypt registered [ 1220.214053] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1221.117630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.141672] alg: No test for adler32 (adler32-zlib) [ 1222.101145] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1222.392837] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1224.055640] Key type lgssc registered [ 1225.631883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.261265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.306350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.699519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.727315] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1247.202583] LustreError: 45213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.231412] LustreError: 45213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1250.520658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.975473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.238912] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1261.874897] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1267.235788] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1274.341583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1274.983308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.156849] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1282.169762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1290.227585] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1296.864749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1296.874057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.887808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.892858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.894074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.896451] Lustre: Skipped 1 previous similar message [ 1302.086507] Lustre: lustre-OST0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 1302.096370] Lustre: Skipped 1 previous similar message [ 1302.636705] Lustre: server umount lustre-OST0000 complete [ 1322.464603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.481815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.509739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.634541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.643796] Lustre: Skipped 5 previous similar messages [ 1335.263308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1335.669785] Lustre: server umount lustre-MDT0000 complete [ 1339.763893] LustreError: 45192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478651 with bad export cookie 1174386663850528543 [ 1339.784743] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.795644] LustreError: 45192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1354.209861] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1354.938153] Lustre: server umount lustre-MDT0001 complete [ 1367.624401] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1370.664401] Key type lgssc unregistered [ 1371.075912] LNet: 48927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.099714] LNetError: 48927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.113583] LNet: Removed LNI 192.168.201.118@tcp [ 1372.193541] Key type .llcrypt unregistered [ 1372.195509] Key type ._llcrypt unregistered [ 1398.111429] Key type ._llcrypt registered [ 1398.113416] Key type .llcrypt registered [ 1398.221654] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1399.223958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.333783] alg: No test for adler32 (adler32-zlib) [ 1400.361473] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1400.673534] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1402.319163] Key type lgssc registered [ 1403.420397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.855395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.882770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.496463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.953904] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1422.821231] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.837632] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1427.938177] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.969750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.312071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.828334] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1440.304944] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.721561] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.450058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.760702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.857815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1461.406156] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1469.820202] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1479.666341] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1484.053025] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1497.147981] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-15 9h26m47s [ 1498.840905] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1502.806163] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 1503.882723] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 1504.226561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.251562] Lustre: Skipped 1 previous similar message [ 1505.342844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.359933] Lustre: Skipped 5 previous similar messages [ 1507.000858] Lustre: server umount lustre-MDT0000 complete [ 1509.962112] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.809277] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1516.109328] LustreError: 52343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.132363] LustreError: 52343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1518.155410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.386128] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.761781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.837775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1519.823526] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1524.627629] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1528.695907] Lustre: DEBUG MARKER: Wait for df (36716) ... [ 1530.645253] Lustre: DEBUG MARKER: done [ 1539.555079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1539.571248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.587809] Lustre: Skipped 1 previous similar message [ 1539.592324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1540.065478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1542.123225] Lustre: server umount lustre-OST0000 complete [ 1550.819050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.821559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1550.836639] Lustre: Skipped 1 previous similar message [ 1550.840514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.840523] Lustre: Skipped 2 previous similar messages [ 1552.763890] Lustre: server umount lustre-MDT0000 complete [ 1558.010536] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.059384] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1559.302310] LustreError: 50324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478871 with bad export cookie 9778572809779040044 [ 1559.327882] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1559.343770] LustreError: 50324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1559.935561] Lustre: server umount lustre-MDT0001 complete [ 1572.545289] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1575.131246] Key type lgssc unregistered [ 1575.421195] LNet: 55273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.431381] LNetError: 55273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.449386] LNet: Removed LNI 192.168.201.118@tcp [ 1576.418625] Key type .llcrypt unregistered [ 1576.422577] Key type ._llcrypt unregistered [ 1627.117647] Key type ._llcrypt registered [ 1627.119842] Key type .llcrypt registered [ 1627.266432] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1628.407058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.600929] alg: No test for adler32 (adler32-zlib) [ 1629.758383] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1630.034789] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1631.761639] Key type lgssc registered [ 1633.437213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1641.631600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1641.655255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.986612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1643.022158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1643.197953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.856978] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1658.516327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.577743] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1658.611845] Lustre: 57487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1658.764327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1662.407895] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1671.194565] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1677.186709] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.347469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.608153] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1686.905974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.833894] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1694.922971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1702.392333] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1708.861442] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-15 9h30m20s [ 1711.123793] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1719.975577] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1723.235532] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1723.330454] LustreError: 59169:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1743.471773] Lustre: lustre-MDT0000: Client 1a291656-0cc8-4fa0-baad-ff2be97b2abe (at 192.168.201.18@tcp) reconnecting [ 1763.375242] LustreError: 59169:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1766.073083] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1768.820766] Lustre: DEBUG MARKER: done [ 1777.126458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.137865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.144389] LustreError: Skipped 1 previous similar message [ 1777.149965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.175185] Lustre: Skipped 1 previous similar message [ 1777.204639] Lustre: Skipped 1 previous similar message [ 1780.204130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.223056] Lustre: Skipped 1 previous similar message [ 1781.597661] Lustre: server umount lustre-OST0000 complete [ 1786.876563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1786.886479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.913968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.972521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.988248] Lustre: Skipped 3 previous similar messages [ 1792.547859] Lustre: server umount lustre-MDT0000 complete [ 1796.907846] LustreError: 57477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479109 with bad export cookie 11263224436066188923 [ 1796.909138] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.921101] LustreError: 57477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1797.635283] Lustre: server umount lustre-MDT0001 complete [ 1809.782787] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1812.762402] Key type lgssc unregistered [ 1813.162946] LNet: 61819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.167410] LNetError: 61819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.187434] LNet: Removed LNI 192.168.201.118@tcp [ 1814.163474] Key type .llcrypt unregistered [ 1814.165594] Key type ._llcrypt unregistered [ 1862.672745] Key type ._llcrypt registered [ 1862.675042] Key type .llcrypt registered [ 1862.796429] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1864.128717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.173293] alg: No test for adler32 (adler32-zlib) [ 1865.430884] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 1865.668534] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1867.327206] Key type lgssc registered [ 1868.747631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1872.355258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1872.393946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.599378] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1873.627177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1873.792416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.855056] Lustre: Failing over lustre-MDT0000 [ 1875.131238] Lustre: server umount lustre-MDT0000 complete [ 1881.197078] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:33:12 (1789479192) [ 1888.695795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.148190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.970732] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1903.245335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.321812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1903.363428] Lustre: 64619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1903.584408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1908.472566] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1916.517376] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1922.604644] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.771724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.947257] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1930.229775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.334341] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1936.992851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1942.781599] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.510682] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1954.786388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1954.793765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.806520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.348724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.350898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.361816] Lustre: Skipped 1 previous similar message [ 1957.381058] Lustre: Skipped 2 previous similar messages [ 1959.608256] Lustre: server umount lustre-MDT0000 complete [ 1962.967230] LustreError: 64612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479275 with bad export cookie 17471126334211572005 [ 1962.980733] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.324099] Lustre: server umount lustre-MDT0001 complete [ 1966.463347] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1971.185819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1979.440780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.943409] Lustre: 63542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789479279/real 1789479279] req@00000000dadddea1 x1876404917543168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789479295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1982.979917] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.128724] Lustre: 63542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789479284/real 1789479284] req@000000003a589203 x1876404917543296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789479300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.740219] LustreError: 66469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.769481] LustreError: 66469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1988.911607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.059054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1989.878062] Lustre: 63541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789479285/real 1789479285] req@000000001010c2ee x1876404917543552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789479301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1992.936020] LustreError: 66470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.959433] LustreError: 68388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.730844] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1994.990183] Lustre: 63540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789479291/real 1789479291] req@00000000279b4d34 x1876404917543680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789479307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1999.330866] LustreError: 66469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.366896] LustreError: 66469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2002.086022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.591861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2006.530970] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2008.071487] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2012.202401] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2015.862893] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.424980] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2027.493347] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.494640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.516326] Lustre: Skipped 1 previous similar message [ 2029.960135] Lustre: server umount lustre-MDT0000 complete [ 2032.610411] LustreError: 69454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.634074] LustreError: 69454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2034.630199] LustreError: 64610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479346 with bad export cookie 17471126334211574280 [ 2034.643775] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2034.648787] LustreError: 64610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2034.968936] Lustre: server umount lustre-MDT0001 complete [ 2042.506137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2052.151633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.023505] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789479349/real 1789479349] req@00000000c5cb223c x1876404917569664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789479365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2053.053544] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.068792] Lustre: Skipped 1 previous similar message [ 2059.232112] LustreError: 63539:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000284914e x1876404917570688/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2059.708754] LustreError: 66469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.758792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.788305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2063.672253] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2065.823461] Lustre: 63542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789479361/real 1789479361] req@0000000006e055cb x1876404917570304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789479377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.860859] Lustre: 63542:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2073.753413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.082202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2074.089351] Lustre: Skipped 2 previous similar messages [ 2079.533851] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2079.718735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2082.833682] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2088.447178] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2095.092459] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2103.277428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.303828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.304894] Lustre: Skipped 2 previous similar messages [ 2103.322542] Lustre: Skipped 2 previous similar messages [ 2108.604681] Lustre: server umount lustre-MDT0000 complete [ 2113.034974] LustreError: 66475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479425 with bad export cookie 17471126334211575659 [ 2113.070696] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2113.508650] LustreError: 71071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.526392] LustreError: 71071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2113.861064] Lustre: server umount lustre-MDT0001 complete [ 2121.046693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2124.758111] Lustre: DEBUG MARKER: files compared the same [ 2126.851649] Lustre: server umount lustre-OST0000 complete [ 2144.997306] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2148.154779] Key type lgssc unregistered [ 2148.527742] LNet: 74357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.536332] LNetError: 74357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.552197] LNet: Removed LNI 192.168.201.118@tcp [ 2149.745152] Key type .llcrypt unregistered [ 2149.749516] Key type ._llcrypt unregistered [ 2178.075787] Key type ._llcrypt registered [ 2178.080752] Key type .llcrypt registered [ 2178.231806] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2179.603609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.658920] alg: No test for adler32 (adler32-zlib) [ 2180.777800] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2181.154476] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2182.944920] Key type lgssc registered [ 2184.271616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2193.545618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2193.586678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.292701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.723591] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug +malloc all [ 2205.664810] LustreError: 75776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.701367] LustreError: 75776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2210.460685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.772793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.743718] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug +malloc all [ 2223.345315] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2228.627797] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2236.314530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2236.715977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.813701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2243.447431] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug +malloc all [ 2253.526962] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2261.471859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2261.474232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.484925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.569105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2265.583335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.587044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.638858] Lustre: Skipped 1 previous similar message [ 2267.561168] Lustre: server umount lustre-OST0000 complete [ 2272.227727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.245177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.256236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.813328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.814508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.400958] Lustre: server umount lustre-MDT0000 complete [ 2280.934540] LustreError: 75775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.955865] LustreError: 75775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2281.312720] LustreError: 75755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479593 with bad export cookie 11449043221236126632 [ 2281.319663] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.332114] LustreError: 75755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2281.674337] Lustre: server umount lustre-MDT0001 complete [ 2289.251338] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2291.949636] Key type lgssc unregistered [ 2292.204797] LNet: 79312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.209994] LNetError: 79312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.222909] LNet: Removed LNI 192.168.201.118@tcp [ 2293.037220] Key type .llcrypt unregistered [ 2293.038917] Key type ._llcrypt unregistered [ 2317.116693] Key type ._llcrypt registered [ 2317.119606] Key type .llcrypt registered [ 2317.292857] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2319.070085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.111259] alg: No test for adler32 (adler32-zlib) [ 2320.289128] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2320.648116] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2322.431575] Key type lgssc registered [ 2324.104524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.564147] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2333.580627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2349.215457] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2354.336272] LustreError: 80704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 2381.024257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.807681] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2395.954524] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2404.275374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.470859] Lustre: *** cfs_fail_loc=706, val=0*** [ 2404.756103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.899557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2409.833693] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2409.959971] LustreError: 80730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.976347] LustreError: 80730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2415.075432] LustreError: 81791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.094933] LustreError: 81791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2418.209534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.573990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2422.817945] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2429.932722] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2435.525252] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2439.137082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.145579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.151019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.158174] Lustre: Skipped 1 previous similar message [ 2443.972628] Lustre: server umount lustre-OST0000 complete [ 2449.379172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.383459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.396970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.398449] Lustre: Skipped 1 previous similar message [ 2449.422517] Lustre: Skipped 2 previous similar messages [ 2454.502559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2454.517440] Lustre: Skipped 1 previous similar message [ 2459.632937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.640065] Lustre: Skipped 1 previous similar message [ 2462.175124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2462.382282] Lustre: server umount lustre-MDT0000 complete [ 2464.736417] LustreError: 81795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.762824] LustreError: 81795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2466.314885] LustreError: 80712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479778 with bad export cookie 7615562742401897045 [ 2466.317729] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.323215] LustreError: 80712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2466.613809] Lustre: server umount lustre-MDT0001 complete [ 2477.801452] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2480.498597] Key type lgssc unregistered [ 2480.756619] LNet: 84324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.764516] LNetError: 84324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.781222] LNet: Removed LNI 192.168.201.118@tcp [ 2481.529410] Key type .llcrypt unregistered [ 2481.530625] Key type ._llcrypt unregistered [ 2504.999105] Key type ._llcrypt registered [ 2505.001650] Key type .llcrypt registered [ 2505.101384] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2505.936537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.965163] alg: No test for adler32 (adler32-zlib) [ 2507.015503] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2507.229530] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2508.952104] Key type lgssc registered [ 2509.965421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2517.524256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2517.556699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.743911] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2529.665242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.083562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.154324] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2535.392246] LustreError: 86422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.427560] LustreError: 86422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2540.515495] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.533470] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2542.396710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2542.657127] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.679695] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2542.705513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.682514] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.697797] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2549.224165] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2552.809964] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.825737] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2558.083143] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.101846] Lustre: MGS: Client b5fbece4-f32c-4975-a7f5-2c55ad855a85 (at 0@lo) reconnecting [ 2558.120561] LustreError: 86436:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2558.512525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.619611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2564.789228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2578.915489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.917689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.931066] Lustre: Skipped 1 previous similar message [ 2578.931352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.957739] Lustre: Skipped 1 previous similar message [ 2584.055210] LustreError: 87150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.083492] LustreError: 87150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2584.292585] Lustre: server umount lustre-OST0000 complete [ 2589.155984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.164982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.175222] Lustre: Skipped 1 previous similar message [ 2589.188319] Lustre: Skipped 1 previous similar message [ 2594.123871] Lustre: server umount lustre-MDT0000 complete [ 2597.878678] Lustre: server umount lustre-MDT0001 complete [ 2601.857372] Lustre: server umount MGS complete [ 2612.993571] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:45:24 (1789479924) [ 2647.458514] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2658.952828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2699.512421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.927308] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2715.527662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.821548] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2715.890587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2715.999692] Lustre: lustre-MDT0001: new disk, initializing [ 2716.087909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.144099] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2716.154511] Lustre: 93593:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2716.164316] LustreError: 93593:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2721.492531] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2732.797161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.020424] Lustre: lustre-OST0000: new disk, initializing [ 2733.023555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2733.035242] Lustre: 94360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2733.124166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.445749] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2740.833371] Lustre: 94777:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000045a294dc x1876405590728960/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789480069 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2749.406281] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2757.107179] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.121532] Lustre: MGS: Client f70ee764-c573-4758-9196-1412b48cb071 (at 0@lo) reconnecting [ 2757.184912] Lustre: 95406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2757.215075] LustreError: 93579:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2757.259473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2757.381533] Lustre: lustre-MDT0000: new disk, initializing [ 2757.613770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.655511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2757.691762] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2758.756421] Lustre: 95430:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000016f921b6 x1876405590735872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789480086 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2763.819358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2763.850319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2763.880425] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2763.930262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2789.857876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.864103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.893628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.418501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.422259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.446775] Lustre: Skipped 1 previous similar message [ 2795.613968] Lustre: server umount lustre-OST0000 complete [ 2802.659036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.670953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.685870] Lustre: Skipped 1 previous similar message [ 2806.048995] Lustre: server umount lustre-MDT0000 complete [ 2807.776552] LustreError: 93568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.803190] LustreError: 93568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2816.094060] Lustre: server umount lustre-MDT0001 complete [ 2819.748718] Lustre: server umount MGS complete [ 2830.845454] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:49:02 (1789480142) [ 2850.524994] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2853.910642] Key type lgssc unregistered [ 2854.211645] LNet: 97936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.218208] LNetError: 97936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.234631] LNet: Removed LNI 192.168.201.118@tcp [ 2855.242826] Key type .llcrypt unregistered [ 2855.244713] Key type ._llcrypt unregistered [ 2871.917368] Key type ._llcrypt registered [ 2871.918922] Key type .llcrypt registered [ 2872.029309] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2873.571151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2873.600052] alg: No test for adler32 (adler32-zlib) [ 2874.854790] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 2875.390626] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2877.135799] Key type lgssc registered [ 2878.493801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2888.788709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.816338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.214097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.657640] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2900.451798] LustreError: 99158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.464622] LustreError: 99158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2905.574140] LustreError: 99157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.460593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.905399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.412258] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2921.426257] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2928.244806] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2936.717649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.259318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.642770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2954.768380] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2960.828777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2971.106516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.115420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.130052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.149950] Lustre: Skipped 1 previous similar message [ 2971.183719] Lustre: Skipped 1 previous similar message [ 2976.226780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.237833] Lustre: Skipped 1 previous similar message [ 2981.347525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2981.358925] Lustre: Skipped 1 previous similar message [ 2982.367197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2982.704308] Lustre: server umount lustre-OST0000 complete [ 2989.024478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2989.032157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.040391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.585856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.965277] Lustre: server umount lustre-MDT0000 complete [ 2996.705585] LustreError: 101411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.721677] LustreError: 101411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2997.257751] LustreError: 99138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480309 with bad export cookie 2712629683509855190 [ 2997.260963] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.281898] LustreError: 99138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2997.913719] Lustre: server umount lustre-MDT0001 complete [ 3010.413519] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3013.517627] Key type lgssc unregistered [ 3013.797742] LNet: 102716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3013.807041] LNetError: 102716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3013.841129] LNet: Removed LNI 192.168.201.118@tcp [ 3014.968417] Key type .llcrypt unregistered [ 3014.973193] Key type ._llcrypt unregistered [ 3034.280451] Key type ._llcrypt registered [ 3034.284505] Key type .llcrypt registered [ 3034.447225] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3035.957811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.115533] alg: No test for adler32 (adler32-zlib) [ 3037.319560] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 3037.686899] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3039.456221] Key type lgssc registered [ 3040.997319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3048.685620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3048.698526] LustreError: 103794:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3048.751732] LustreError: 103801:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3048.756977] LustreError: 103794:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4953 [ 3048.770058] LustreError: 103794:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3048.778290] LustreError: 103794:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3048.791823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.634627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.263823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.662237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3065.550744] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3071.817832] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing lsmod [ 3074.910228] LustreError: 105007:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3074.959445] LustreError: 105012:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3074.965340] LustreError: 105007:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4957 [ 3074.970262] LustreError: 105007:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3074.976604] LustreError: 105007:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3074.987117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3075.446181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.160626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3080.181817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.199451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.704110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.718634] Lustre: Skipped 1 previous similar message [ 3084.796676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.436218] Lustre: server umount lustre-MDT0000 complete [ 3089.797927] LustreError: 103820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480401 with bad export cookie 2966653870695359762 [ 3089.800469] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.810104] LustreError: 103820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3089.888403] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.890734] LustreError: 103835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.912114] Lustre: Skipped 2 previous similar messages [ 3089.918091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3089.943025] LustreError: 103835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3089.984880] Lustre: Skipped 2 previous similar messages [ 3091.039465] LustreError: 104271:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003f6e5cb7 x1876406146454144/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3091.302115] Lustre: server umount lustre-MDT0001 complete [ 3095.887221] Lustre: server umount lustre-OST0000 complete [ 3104.972704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.439257] LustreError: 106076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.517965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.164188] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3110.880353] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.980075] LustreError: 106076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.334860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.793669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.158795] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3133.362236] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3139.830811] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3147.247239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3147.611666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.723572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3154.287805] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3162.355616] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3176.417773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3176.428839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.445201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3179.488747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3179.505414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.430426] Lustre: server umount lustre-OST0000 complete [ 3186.659915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3186.674174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.688868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3186.693471] Lustre: Skipped 2 previous similar messages [ 3191.696996] Lustre: server umount lustre-MDT0000 complete [ 3194.854321] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.874240] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3195.739168] LustreError: 106057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480507 with bad export cookie 2966653870695360462 [ 3195.757279] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.380901] Lustre: server umount lustre-MDT0001 complete [ 3208.497867] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3211.931698] Key type lgssc unregistered [ 3212.313534] LNet: 109625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3212.320099] LNetError: 109625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3212.345845] LNet: Removed LNI 192.168.201.118@tcp [ 3213.778142] Key type .llcrypt unregistered [ 3213.783579] Key type ._llcrypt unregistered [ 3242.747317] Key type ._llcrypt registered [ 3242.750316] Key type .llcrypt registered [ 3242.940080] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3244.275086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3244.372987] alg: No test for adler32 (adler32-zlib) [ 3245.574783] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 3245.897770] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3247.663326] Key type lgssc registered [ 3249.219322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3258.220151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3258.241885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.858528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.797778] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3270.121231] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.133620] LustreError: 111045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3274.774815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.185852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.548411] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3288.798899] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3294.903517] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3302.066427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.437296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.558879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3308.837465] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3317.069139] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3326.317538] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3330.269600] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3335.655405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.658917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.681312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.701401] Lustre: Skipped 1 previous similar message [ 3340.772924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.788948] Lustre: Skipped 1 previous similar message [ 3341.271856] Lustre: server umount lustre-OST0000 complete [ 3345.889613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.900154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.918313] Lustre: Skipped 2 previous similar messages [ 3345.938449] Lustre: Skipped 1 previous similar message [ 3351.015549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.039872] Lustre: Skipped 1 previous similar message [ 3352.139422] Lustre: server umount lustre-MDT0000 complete [ 3356.069064] LustreError: 111026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480668 with bad export cookie 12532929279016098596 [ 3356.106420] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.548449] Lustre: server umount lustre-MDT0001 complete [ 3367.500058] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3370.210861] Key type lgssc unregistered [ 3370.499763] LNet: 114688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3370.506550] LNetError: 114688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3370.538463] LNet: Removed LNI 192.168.201.118@tcp [ 3371.332202] Key type .llcrypt unregistered [ 3371.333890] Key type ._llcrypt unregistered [ 3387.034488] Key type ._llcrypt registered [ 3387.041040] Key type .llcrypt registered [ 3387.132863] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3388.049580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3388.149078] alg: No test for adler32 (adler32-zlib) [ 3389.115614] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 3389.322710] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3390.943175] Key type lgssc registered [ 3392.050745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3398.761770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3398.794278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.329247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.805890] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3409.893269] LustreError: 115910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.912269] LustreError: 115910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3414.085575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.501572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3418.824288] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3425.969421] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3431.220489] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3437.443039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.893838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3443.987101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3445.034354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3451.069036] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3459.552885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.562594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.584225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.582320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.582542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.586125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3465.700614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3465.707734] Lustre: Skipped 2 previous similar messages [ 3470.819104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3470.832659] Lustre: Skipped 1 previous similar message [ 3473.378117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3473.685282] Lustre: server umount lustre-OST0000 complete [ 3481.058811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3481.061195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.066947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3481.071185] Lustre: Skipped 1 previous similar message [ 3481.086018] Lustre: Skipped 1 previous similar message [ 3491.295853] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3491.297076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3491.316086] Lustre: Skipped 3 previous similar messages [ 3491.595479] Lustre: server umount lustre-MDT0000 complete [ 3495.227472] LustreError: 115891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480807 with bad export cookie 8094196353701380696 [ 3495.249560] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.513855] Lustre: server umount lustre-MDT0001 complete [ 3505.753565] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3508.449653] Key type lgssc unregistered [ 3508.775065] LNet: 119449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3508.784845] LNetError: 119449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3508.799382] LNet: Removed LNI 192.168.201.118@tcp [ 3509.617670] Key type .llcrypt unregistered [ 3509.624374] Key type ._llcrypt unregistered [ 3532.350793] Key type ._llcrypt registered [ 3532.352543] Key type .llcrypt registered [ 3532.499240] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3533.944983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3533.981669] alg: No test for adler32 (adler32-zlib) [ 3535.196538] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 3535.539165] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3537.287161] Key type lgssc registered [ 3538.443708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3545.804346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3545.833038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.555860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.012777] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3556.846646] LustreError: 120865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.874121] LustreError: 120865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3561.964698] LustreError: 120866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3562.010124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.466956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.137142] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3573.813527] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3579.289857] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3586.424388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.818992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.930559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3592.488730] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3598.767357] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3605.429248] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3608.385099] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3608.389981] Lustre: Skipped 2 previous similar messages [ 3612.786917] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3612.796044] Lustre: Skipped 2 previous similar messages [ 3613.785899] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3613.813725] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3619.063691] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.18@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3619.070358] Lustre: Skipped 2 previous similar messages [ 3622.665898] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.18@tcp 192.168.0.[2,10]@tcp [ 3626.510344] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.18/24@tcp in log lustre-MDT0000 [ 3626.519900] Lustre: Skipped 2 previous similar messages [ 3629.901684] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.18/24@tcp [ 3629.907128] Lustre: Skipped 1 previous similar message [ 3636.719775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3636.727778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.749346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.367637] Lustre: server umount lustre-OST0000 complete [ 3644.385862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3644.394673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.406957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.433413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.442484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.514235] Lustre: server umount lustre-MDT0000 complete [ 3651.554287] LustreError: 121884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.565312] LustreError: 121884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3651.766555] LustreError: 121613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480963 with bad export cookie 15873238958609866910 [ 3651.769953] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.775820] LustreError: 121613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3661.454310] Lustre: server umount lustre-MDT0001 complete [ 3672.242067] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3674.634594] Key type lgssc unregistered [ 3674.874659] LNet: 125669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3674.879457] LNetError: 125669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3674.890251] LNet: Removed LNI 192.168.201.118@tcp [ 3675.586906] Key type .llcrypt unregistered [ 3675.589565] Key type ._llcrypt unregistered [ 3702.464886] Key type ._llcrypt registered [ 3702.467539] Key type .llcrypt registered [ 3702.636728] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3704.098531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3704.173088] alg: No test for adler32 (adler32-zlib) [ 3705.200154] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 3705.480930] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3707.143150] Key type lgssc registered [ 3708.528514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.819842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.827900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.222651] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3717.254154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.263554] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.18/24@tcp [ 3722.032634] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3726.817875] LustreError: 127232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.842494] LustreError: 127232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3729.492920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.819532] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3729.843723] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.18/24@tcp [ 3729.888903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.552899] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3740.319536] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.206852] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3750.960594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3751.175731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.063595] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3762.338042] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3762.499171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3772.902943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3772.903651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.916872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3772.920952] Lustre: Skipped 1 previous similar message [ 3772.931513] Lustre: Skipped 1 previous similar message [ 3777.615954] Lustre: server umount lustre-OST0000 complete [ 3781.110384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3781.117921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.125606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3783.138051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3783.138949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.682345] Lustre: server umount lustre-MDT0000 complete [ 3788.257327] LustreError: 127231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.271398] LustreError: 127231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3789.679810] LustreError: 127213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481101 with bad export cookie 5713331604405082065 [ 3789.681037] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.686088] LustreError: 127213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3789.896381] Lustre: server umount lustre-MDT0001 complete [ 3798.916517] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3801.228676] Key type lgssc unregistered [ 3801.481983] LNet: 130820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3801.496329] LNetError: 130820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3801.534778] LNet: Removed LNI 192.168.201.118@tcp [ 3802.274838] Key type .llcrypt unregistered [ 3802.278261] Key type ._llcrypt unregistered [ 3826.766378] Key type ._llcrypt registered [ 3826.771470] Key type .llcrypt registered [ 3826.898906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3840.696623] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3841.905167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3841.924921] alg: No test for adler32 (adler32-zlib) [ 3842.960567] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 3843.245526] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3844.895159] Key type lgssc registered [ 3845.966235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3876.894180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3876.920676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.161613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3878.203311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3878.289676] Lustre: lustre-MDT0000: new disk, initializing [ 3878.374087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.389397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3882.926629] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3892.560695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.716544] Lustre: 134213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3892.770753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3892.775170] Lustre: Skipped 1 previous similar message [ 3892.849707] Lustre: lustre-MDT0001: new disk, initializing [ 3892.968649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3893.041406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3893.056176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3897.747506] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3906.418244] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3910.975369] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3916.547558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.734460] Lustre: lustre-OST0000: new disk, initializing [ 3916.738201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3916.743889] Lustre: 136179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3916.817719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.462171] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3926.055897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3926.078084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3926.175872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3931.153565] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3935.405398] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3935.601504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3939.922421] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3940.105395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3941.857163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3941.859606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.872976] LustreError: Skipped 1 previous similar message [ 3941.889473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.984387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.987201] Lustre: Skipped 1 previous similar message [ 3947.826608] Lustre: server umount lustre-OST0000 complete [ 3952.102509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.115308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3952.127733] Lustre: Skipped 2 previous similar messages [ 3957.216970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.228877] Lustre: Skipped 2 previous similar messages [ 3957.393255] Lustre: server umount lustre-MDT0000 complete [ 3960.399687] LustreError: 134204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481272 with bad export cookie 9444149620550372231 [ 3960.402509] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.411302] LustreError: 134204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3960.627388] Lustre: server umount lustre-MDT0001 complete [ 3967.254545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.593994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.548479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3973.089759] LustreError: 138394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.100030] LustreError: 138394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3977.189247] LustreError: 138393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3978.184525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.413177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3982.134200] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3987.732298] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3992.146343] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3996.909478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.272516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.526844] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4008.735962] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4012.522977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4013.773185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4017.507952] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4017.716112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4025.905393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.125265] Lustre: lustre-OST0001: new disk, initializing [ 4026.131293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4026.144785] Lustre: 141769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4026.236748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4032.020492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4032.568970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4032.576093] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4032.682705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4039.119813] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4039.306451] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4043.234109] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4043.432118] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4048.325586] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e3f8288e800.ost_server_uuid 50 [ 4049.857396] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e3f8288e800.ost_server_uuid in FULL state after 0 sec [ 4058.607489] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.613241] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4058.624054] LustreError: Skipped 1 previous similar message [ 4058.625098] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4058.632680] Lustre: Skipped 1 previous similar message [ 4062.622877] Lustre: server umount lustre-OST0001 complete [ 4063.719192] LustreError: 140214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.747624] LustreError: 140214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4068.833273] LustreError: 140212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.839870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.850943] LustreError: 140212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4068.874783] Lustre: Skipped 1 previous similar message [ 4068.879166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4068.886939] Lustre: Skipped 1 previous similar message [ 4071.486172] Lustre: server umount lustre-OST0000 complete [ 4076.001166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4076.005216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.010764] Lustre: Skipped 1 previous similar message [ 4080.769766] Lustre: server umount lustre-MDT0000 complete [ 4083.549874] LustreError: 138373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481395 with bad export cookie 9444149620550373113 [ 4083.560119] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4083.843275] Lustre: server umount lustre-MDT0001 complete [ 4099.122139] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4101.255107] Key type lgssc unregistered [ 4101.536128] LNet: 144773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4101.543965] LNetError: 144773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4101.556928] LNet: Removed LNI 192.168.201.118@tcp [ 4102.306321] Key type .llcrypt unregistered [ 4102.311965] Key type ._llcrypt unregistered [ 4133.219354] Key type ._llcrypt registered [ 4133.220439] Key type .llcrypt registered [ 4133.293377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4143.785938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4144.590229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4144.615212] alg: No test for adler32 (adler32-zlib) [ 4145.707446] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 4145.892911] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4147.575411] Key type lgssc registered [ 4148.611648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4173.737772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4173.752171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.946294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4174.973313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4175.053196] Lustre: lustre-MDT0000: new disk, initializing [ 4175.116833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.129601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4178.512628] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4187.208933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.301428] Lustre: 148904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4187.356259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4187.360975] Lustre: Skipped 1 previous similar message [ 4187.440435] Lustre: lustre-MDT0001: new disk, initializing [ 4187.511173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.545136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4187.557583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4191.197816] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4198.492876] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4203.079618] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4208.925319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.133568] Lustre: lustre-OST0000: new disk, initializing [ 4209.138321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4209.144789] Lustre: 150869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4209.220873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.921519] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4216.899885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4216.907335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4217.016105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4222.545973] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4226.793584] Lustre: Failing over lustre-OST0000 [ 4226.912267] Lustre: server umount lustre-OST0000 complete [ 4242.911140] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481539/real 1789481539] req@00000000f1566938 x1876407309334912/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789481555 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4242.915541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.931243] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4242.949184] Lustre: Skipped 1 previous similar message [ 4243.705715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4243.921698] Lustre: 147127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481539/real 1789481539] req@000000007ec38965 x1876407309335808/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789481555 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4243.949518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.965308] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4245.606158] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4245.990806] Lustre: 147129:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481542/real 1789481542] req@00000000ecb6303f x1876407309336192/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789481558 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4248.015978] Lustre: 147128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481544/real 1789481544] req@0000000088a89f29 x1876407309336960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789481560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4249.329187] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4252.205304] Lustre: Failing over lustre-MDT0000 [ 4252.512504] Lustre: server umount lustre-MDT0000 complete [ 4253.151777] Lustre: 147130:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481549/real 1789481549] req@000000005312bbf2 x1876407309338112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789481565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4253.153550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.174546] Lustre: 147130:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4253.198716] Lustre: Skipped 2 previous similar messages [ 4259.297544] LustreError: 148913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.315876] LustreError: 148913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4260.947922] LustreError: 149293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.418452] LustreError: 148911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.432406] LustreError: 148911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4269.024727] Lustre: 148908:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481565/real 1789481565] req@000000005312bbf2 x1876407309346176/t0(0) o101->MGC192.168.201.118@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1789481581 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4269.053254] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4269.065166] Lustre: 148908:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4269.072125] LustreError: 151392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.083563] LustreError: 151392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4269.360353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.151704] LustreError: 148913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.168249] LustreError: 148913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4279.411525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4279.447781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.497689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4280.415763] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4283.539680] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4284.902720] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4284.932683] Lustre: lustre-OST0000: Recovery over after 0:39, of 3 clients 3 recovered and 0 were evicted. [ 4284.986530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4295.146921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4295.147159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.156203] LustreError: Skipped 1 previous similar message [ 4295.174772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4296.794967] Lustre: server umount lustre-OST0000 complete [ 4300.257581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4300.261960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.269869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4300.269876] Lustre: Skipped 1 previous similar message [ 4300.320672] Lustre: Skipped 2 previous similar messages [ 4305.381907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.392414] Lustre: Skipped 2 previous similar messages [ 4305.792883] Lustre: server umount lustre-MDT0000 complete [ 4309.249233] LustreError: 148898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481621 with bad export cookie 58981465100512508 [ 4309.251141] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.266335] LustreError: 148898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4309.690434] Lustre: server umount lustre-MDT0001 complete [ 4319.133101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4321.259113] Key type lgssc unregistered [ 4321.492657] LNet: 154528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4321.498390] LNetError: 154528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4321.509890] LNet: Removed LNI 192.168.201.118@tcp [ 4322.313219] Key type .llcrypt unregistered [ 4322.316409] Key type ._llcrypt unregistered [ 4353.894243] Key type ._llcrypt registered [ 4353.896548] Key type .llcrypt registered [ 4353.967696] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4354.852475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4354.976068] alg: No test for adler32 (adler32-zlib) [ 4355.982601] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 4356.204167] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4357.863277] Key type lgssc registered [ 4358.791803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4366.109577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4366.122567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.550325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.349487] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4377.063450] LustreError: 156047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.085090] LustreError: 156047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4378.991922] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4382.184738] LustreError: 156046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.224621] LustreError: 156075:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4382.234285] LustreError: 156075:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4382.247202] LustreError: 156075:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4382.449850] Lustre: server umount lustre-MDT0000 complete [ 4395.274897] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4403.688719] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4432.108970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.251640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4432.287822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4432.379059] Lustre: lustre-MDT0000: new disk, initializing [ 4432.471287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.484824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4436.211577] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4445.015268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.146226] Lustre: 159377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4445.212500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4445.216184] Lustre: Skipped 1 previous similar message [ 4445.288401] Lustre: lustre-MDT0001: new disk, initializing [ 4445.371099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4445.404136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4445.423519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4448.609300] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4458.373281] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4468.715710] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4478.655414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.168764] Lustre: lustre-OST0000: new disk, initializing [ 4479.172988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4479.197590] Lustre: 161349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4479.443946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.316914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4487.339836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4487.497446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4488.871147] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4502.471529] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4508.554758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4511.486103] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:17:02 (1789481822) [ 4757.987641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4758.001207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.017561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4759.012471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4759.013833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.129581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4764.137101] Lustre: Skipped 4 previous similar messages [ 4769.247137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4769.255531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.268159] Lustre: Skipped 2 previous similar messages [ 4769.489446] Lustre: server umount lustre-MDT0000 complete [ 4773.672549] LustreError: 161348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482085 with bad export cookie 15879020964668626144 [ 4773.672917] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4773.685030] LustreError: 161348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4774.374170] LustreError: 159384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.381865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.427508] Lustre: Skipped 1 previous similar message [ 4774.433822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4778.401547] LustreError: 159383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.079322] Lustre: server umount lustre-MDT0001 complete [ 4790.573326] Lustre: server umount lustre-OST0000 complete [ 4801.508644] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4813.552536] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4854.330086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.539395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4854.562174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4854.632515] Lustre: lustre-MDT0000: new disk, initializing [ 4854.732385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.750390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4859.545352] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4870.687736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.851607] Lustre: 166012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4871.012331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4871.024702] Lustre: Skipped 1 previous similar message [ 4871.180618] Lustre: lustre-MDT0001: new disk, initializing [ 4871.397309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4871.460945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4871.481207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4876.686807] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4887.151932] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4893.698291] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4901.065862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4901.393529] Lustre: lustre-OST0000: new disk, initializing [ 4901.398590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4901.406446] Lustre: 167981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4901.483842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.597694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4907.607884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4907.783212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4908.422828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4918.684779] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4924.031202] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4924.273074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4929.428542] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4929.658564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4933.602383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.612914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.614284] LustreError: Skipped 1 previous similar message [ 4933.616231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.616238] Lustre: Skipped 1 previous similar message [ 4933.663236] Lustre: Skipped 1 previous similar message [ 4937.706628] Lustre: server umount lustre-OST0000 complete [ 4943.330192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4943.334236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.641490] Lustre: server umount lustre-MDT0000 complete [ 4948.961721] LustreError: 168281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.993931] LustreError: 168281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4952.023093] LustreError: 166003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482264 with bad export cookie 15879020964668784974 [ 4952.025424] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4952.039461] LustreError: 166003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4952.442242] Lustre: server umount lustre-MDT0001 complete [ 4961.335562] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:24:32 (1789482272) [ 4977.653067] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4988.848277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5024.840332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.073224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5025.121474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5025.231502] Lustre: lustre-MDT0000: new disk, initializing [ 5025.305969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5025.320354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5030.410327] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5042.024913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.227911] Lustre: 172735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5042.542642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5047.578605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5055.914773] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5061.208824] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.103349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.466304] Lustre: lustre-OST0000: new disk, initializing [ 5071.469413] Lustre: Skipped 1 previous similar message [ 5071.475421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5071.480726] Lustre: Skipped 2 previous similar messages [ 5071.485747] Lustre: 174715:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5079.128684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5079.138219] Lustre: Skipped 1 previous similar message [ 5079.149896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5079.278194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5079.339081] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5088.668943] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5100.013435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5100.029703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.039846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5100.039855] Lustre: Skipped 3 previous similar messages [ 5100.071924] Lustre: Skipped 2 previous similar messages [ 5103.328836] Lustre: server umount lustre-OST0000 complete [ 5109.216047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5109.221417] LustreError: Skipped 1 previous similar message [ 5113.725275] Lustre: server umount lustre-MDT0000 complete [ 5115.363284] LustreError: 172747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.388741] LustreError: 172747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5117.434025] LustreError: 172728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482429 with bad export cookie 15879020964668785856 [ 5117.443566] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.451373] LustreError: 172728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5117.984646] Lustre: server umount lustre-MDT0001 complete [ 5126.727194] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:27:18 (1789482438) [ 5144.874355] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 5156.855781] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5199.678025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.894799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5199.916942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5199.999907] Lustre: lustre-MDT0000: new disk, initializing [ 5200.124377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5200.132209] Lustre: Skipped 2 previous similar messages [ 5200.170186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5204.639525] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5215.172734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.417894] Lustre: 179040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5215.802970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5223.207271] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5236.276740] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5240.971418] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5246.407488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5246.696219] Lustre: 181017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5252.280834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5252.449449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5252.754216] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5261.521305] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5267.936838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5267.939706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.951964] LustreError: Skipped 1 previous similar message [ 5267.952517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.952521] Lustre: Skipped 4 previous similar messages [ 5268.002525] Lustre: Skipped 3 previous similar messages [ 5269.320520] Lustre: server umount lustre-OST0000 complete [ 5287.391152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5289.635731] Lustre: server umount lustre-MDT0000 complete [ 5291.493566] LustreError: 179050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5291.524095] LustreError: 179050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5293.590243] LustreError: 179030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482605 with bad export cookie 15879020964668786941 [ 5293.594082] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5293.598734] LustreError: 179030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5294.065179] Lustre: server umount lustre-MDT0001 complete [ 5304.644371] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5307.823519] Key type lgssc unregistered [ 5308.093344] LNet: 182875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5308.103567] LNetError: 182875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5308.118070] LNet: Removed LNI 192.168.201.118@tcp [ 5309.102352] Key type .llcrypt unregistered [ 5309.109814] Key type ._llcrypt unregistered [ 5333.868624] Key type ._llcrypt registered [ 5333.874262] Key type .llcrypt registered [ 5334.029331] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5335.469058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5335.581727] alg: No test for adler32 (adler32-zlib) [ 5336.680374] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 5336.949382] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5338.639169] Key type lgssc registered [ 5339.859974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5346.700978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5346.718900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.220681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.795503] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5358.562037] LustreError: 184301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.575714] LustreError: 184301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5360.983166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.328246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5366.516520] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5374.445533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5380.306673] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5386.453299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5386.725243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.911377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5392.886880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5398.899425] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5423.591785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5423.614356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.630150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.674651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.681904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.682177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.741733] Lustre: server umount lustre-OST0000 complete [ 5432.811060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5432.815135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.835565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5432.842779] Lustre: Skipped 1 previous similar message [ 5436.896805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5436.898823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.016701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5442.020932] Lustre: Skipped 1 previous similar message [ 5447.135360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5447.256629] Lustre: server umount lustre-MDT0000 complete [ 5450.968078] LustreError: 184284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482763 with bad export cookie 5215145617095788879 [ 5450.972826] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5450.976319] LustreError: 184284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5451.252329] Lustre: server umount lustre-MDT0001 complete [ 5463.534448] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5467.078992] Key type lgssc unregistered [ 5467.358987] LNet: 187862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5467.372890] LNetError: 187862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5468.396095] LNet: Removed LNI 192.168.201.118@tcp [ 5469.730960] Key type .llcrypt unregistered [ 5469.738262] Key type ._llcrypt unregistered [ 5494.137241] Key type ._llcrypt registered [ 5494.141428] Key type .llcrypt registered [ 5494.284938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5495.314687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5495.531795] alg: No test for adler32 (adler32-zlib) [ 5496.667326] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 5497.050090] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5498.703600] Key type lgssc registered [ 5500.410255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5508.678491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5508.693658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.264665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.215429] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5519.842277] LustreError: 189279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.870201] LustreError: 189279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5524.581348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.866518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.161659] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5538.632504] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5543.905634] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5552.368883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.862366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.947928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5559.451518] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5567.698377] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5579.869461] Lustre: lustre-OST0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 5580.777248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.777296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.796506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5580.808488] LustreError: Skipped 1 previous similar message [ 5580.824597] Lustre: Skipped 1 previous similar message [ 5582.875787] Lustre: server umount lustre-OST0000 complete [ 5590.401781] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e3f83c8b800.ost_server_uuid 50 [ 5592.159981] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5606.371187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.380188] Lustre: Skipped 1 previous similar message [ 5606.385442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.390145] Lustre: Skipped 1 previous similar message [ 5611.494573] LustreError: 190008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.651982] Lustre: server umount lustre-MDT0000 complete [ 5615.094202] LustreError: 189258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482927 with bad export cookie 1189639552581713082 [ 5615.103954] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.111049] LustreError: 189258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5615.372458] Lustre: server umount lustre-MDT0001 complete [ 5623.476390] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:35:35 (1789482935) [ 5630.555598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.995196] LustreError: 193195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.007562] LustreError: 193195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5631.060255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.471934] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5636.068288] LustreError: 193196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.083517] LustreError: 193196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5641.187220] LustreError: 193195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.022276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.377995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5648.415569] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5655.195718] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5660.481284] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5666.488679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5666.975675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.257269] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5676.209359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5680.304275] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5686.973956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.163347] Lustre: lustre-OST0001: new disk, initializing [ 5687.166577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5687.172019] Lustre: 196034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5687.278432] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.549720] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5694.616367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5694.632040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5694.766735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5702.862503] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5707.235290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.249879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.265729] Lustre: Skipped 1 previous similar message [ 5707.273983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5710.330777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.623149] Lustre: server umount lustre-OST0000 complete [ 5713.991557] LustreError: 195016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5718.444096] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5718.728699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5723.239574] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5723.474331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5730.805087] LustreError: 195014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.853967] LustreError: 195014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5741.026554] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.033819] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5741.043816] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5741.049654] Lustre: Skipped 4 previous similar messages [ 5743.405895] Lustre: server umount lustre-OST0001 complete [ 5751.265112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5751.266867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5751.291260] Lustre: Skipped 2 previous similar messages [ 5753.679750] Lustre: server umount lustre-MDT0000 complete [ 5757.186592] LustreError: 193175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483069 with bad export cookie 1189639552581714153 [ 5757.208959] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5757.678125] Lustre: server umount lustre-MDT0001 complete [ 5780.525858] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:38:11 (1789483091) [ 5787.898603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.467994] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5788.530416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5788.764539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.608454] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5802.905385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.977604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5803.009120] Lustre: 199478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5807.859730] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5815.354882] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.319704] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5828.226389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.516154] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5835.458665] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5836.453181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5842.944632] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5849.671295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5849.716900] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5856.153865] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5857.036323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5864.319478] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.976286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5866.986382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.002364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5867.008425] Lustre: Skipped 1 previous similar message [ 5881.311288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5881.634025] Lustre: server umount lustre-OST0000 complete [ 5882.878987] LustreError: 202751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.906201] LustreError: 202751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5898.213461] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5898.227605] LustreError: Skipped 1 previous similar message [ 5903.343047] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5903.348715] Lustre: Skipped 12 previous similar messages [ 5904.639728] Lustre: server umount lustre-OST0001 complete [ 5911.023694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5911.029043] LustreError: Skipped 1 previous similar message [ 5911.032837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.058523] Lustre: Skipped 3 previous similar messages [ 5914.985756] Lustre: server umount lustre-MDT0000 complete [ 5919.146066] LustreError: 199469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483231 with bad export cookie 1189639552581715553 [ 5919.155681] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5919.157375] LustreError: 199469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5919.549911] Lustre: server umount lustre-MDT0001 complete [ 5943.027598] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:40:54 (1789483254) [ 5961.380502] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 5972.248556] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6008.122379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.331848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6008.358376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6008.466944] Lustre: lustre-MDT0000: new disk, initializing [ 6008.552422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.563473] Lustre: Skipped 3 previous similar messages [ 6008.587081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6013.684908] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6023.068982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.207618] Lustre: 207574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6023.270278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6023.274026] Lustre: Skipped 1 previous similar message [ 6023.374562] Lustre: lustre-MDT0001: new disk, initializing [ 6023.484829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6023.497260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6027.372925] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6036.538859] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6041.678615] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6047.981082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6048.270844] Lustre: lustre-OST0000: new disk, initializing [ 6048.276448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6048.284705] Lustre: 209542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6054.137442] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6057.543768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6057.563845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6057.674334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6063.724502] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6068.364393] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6068.613323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6073.188801] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6073.463633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6078.455036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.466148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.475240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.479132] Lustre: Skipped 2 previous similar messages [ 6078.483219] Lustre: Skipped 3 previous similar messages [ 6081.916793] Lustre: server umount lustre-OST0000 complete [ 6091.976241] Lustre: server umount lustre-MDT0000 complete [ 6093.796225] LustreError: 207580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.827314] LustreError: 207580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6096.335909] LustreError: 207564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483408 with bad export cookie 1189639552581717352 [ 6096.369400] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.371367] LustreError: 207564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6097.149830] Lustre: server umount lustre-MDT0001 complete [ 6106.160795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.226711] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6120.156879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.771697] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6133.816350] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6140.946396] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6149.400082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6149.943217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.956415] Lustre: Skipped 4 previous similar messages [ 6158.546524] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6166.020601] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6170.361372] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6170.547940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6174.788329] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6175.036958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6180.842149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.863932] LustreError: Skipped 1 previous similar message [ 6183.530484] Lustre: server umount lustre-OST0000 complete [ 6190.936272] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6196.124404] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6226.357035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6233.007654] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6240.738811] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.519907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6247.587369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.595766] Lustre: Skipped 5 previous similar messages [ 6248.926411] Lustre: server umount lustre-OST0000 complete [ 6253.027088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.056833] Lustre: Skipped 4 previous similar messages [ 6267.359152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6267.611368] Lustre: server umount lustre-MDT0000 complete [ 6268.386464] LustreError: 211760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.420257] LustreError: 211760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6271.468347] LustreError: 211741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483583 with bad export cookie 1189639552581718234 [ 6271.480813] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.486040] LustreError: 211741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6271.915398] Lustre: server umount lustre-MDT0001 complete [ 6280.714419] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:46:32 (1789483592) [ 6288.163168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.628330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6303.169502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.928512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6317.198528] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6323.959088] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6331.917424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6339.508077] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6347.519859] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6353.727386] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6354.005726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6359.339554] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6359.564857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6366.248876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6366.522800] Lustre: lustre-OST0001: new disk, initializing [ 6366.526710] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6366.534675] Lustre: 221104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6373.015755] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6374.540771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6374.555713] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6374.720430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6383.648758] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6388.311372] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6388.695808] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6394.802397] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6395.107954] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6412.257760] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6412.424491] Lustre: server umount lustre-OST0001 complete [ 6419.615260] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6419.816588] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6424.899121] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6425.122554] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6455.652496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.008800] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6456.011739] Lustre: Skipped 5 previous similar messages [ 6461.442055] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6461.460259] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6461.469865] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6463.145576] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6471.313641] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6472.940191] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6476.768866] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6476.791565] LustreError: Skipped 1 previous similar message [ 6481.258272] Lustre: server umount lustre-OST0001 complete [ 6504.892547] Lustre: server umount lustre-OST0000 complete [ 6512.608545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6512.615441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6512.632922] Lustre: Skipped 8 previous similar messages [ 6512.638522] Lustre: Skipped 17 previous similar messages [ 6523.359135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6523.701894] Lustre: server umount lustre-MDT0000 complete [ 6526.957976] LustreError: 222447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.989312] LustreError: 222447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 6527.489249] LustreError: 217714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483839 with bad export cookie 1189639552581719186 [ 6527.498134] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6527.502257] LustreError: 217714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6534.359764] Lustre: server umount lustre-MDT0001 complete [ 6559.304408] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:51:10 (1789483870) [ 6566.821311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.241085] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6567.289372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6572.227773] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6580.594252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6580.742337] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6580.760493] Lustre: 226349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6585.320634] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6593.333312] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6598.802212] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6605.704206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.907770] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6613.158918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6621.042099] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6631.105862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6631.225874] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6638.485935] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6646.788288] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6648.225240] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6653.040381] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6653.249482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6657.910317] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6658.047860] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6663.506878] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e3f927ed800.ost_server_uuid 50 [ 6664.819715] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e3f927ed800.ost_server_uuid in IDLE state after 0 sec [ 6665.673817] Lustre: Permanently deactivating lustre-OST0001 [ 6665.687871] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6669.429499] Lustre: Permanently reactivating lustre-OST0001 [ 6676.474079] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.486513] LustreError: Skipped 3 previous similar messages [ 6679.694436] Lustre: server umount lustre-OST0001 complete [ 6689.256806] Lustre: server umount lustre-OST0000 complete [ 6699.467377] Lustre: server umount lustre-MDT0000 complete [ 6703.250562] LustreError: 228210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484015 with bad export cookie 1189639552581720474 [ 6703.251207] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.263524] LustreError: 228210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6703.614852] Lustre: server umount lustre-MDT0001 complete [ 6727.121169] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:53:58 (1789484038) [ 6734.724332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.064061] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6735.103854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6735.114351] Lustre: Skipped 5 previous similar messages [ 6740.073259] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6747.833193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.950961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6747.966164] Lustre: 233042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6752.878822] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6760.891812] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6766.616493] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6773.259487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.477990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6773.494925] Lustre: Permanently deactivating lustre-OST0000 [ 6773.505578] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6779.943991] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6783.063604] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6783.164625] LustreError: 235405:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6787.016984] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6793.255215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.350446] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6800.209540] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6801.181104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6807.991316] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6809.683050] Lustre: Permanently reactivating lustre-OST0000 [ 6809.696752] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6809.700263] Lustre: Skipped 2 previous similar messages [ 6815.830313] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6815.831578] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 00000000fa55d049, cur 1789484128 deadline 1789484125 last 1789484095 [ 6815.835455] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.837266] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6815.837273] Lustre: Skipped 1 previous similar message [ 6815.877520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.922510] LustreError: 235405:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -107 [ 6816.996931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6824.716117] Lustre: server umount lustre-OST0001 complete [ 6842.339974] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6842.728663] Lustre: server umount lustre-OST0000 complete [ 6853.781582] Lustre: server umount lustre-MDT0000 complete [ 6857.107357] LustreError: 233034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484169 with bad export cookie 1189639552581722126 [ 6857.112285] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6857.128823] LustreError: 233034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6857.555590] Lustre: server umount lustre-MDT0001 complete [ 6869.063258] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6872.103957] Key type lgssc unregistered [ 6872.286992] LNet: 237926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6872.300602] LNetError: 237926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6872.319364] LNet: Removed LNI 192.168.201.118@tcp [ 6873.107743] Key type .llcrypt unregistered [ 6873.111037] Key type ._llcrypt unregistered [ 6898.444626] Key type ._llcrypt registered [ 6898.446826] Key type .llcrypt registered [ 6898.573513] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6899.691630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6899.761659] alg: No test for adler32 (adler32-zlib) [ 6900.762496] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 6900.991410] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6902.687957] Key type lgssc registered [ 6904.040863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6914.372780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6914.408541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.179697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.454661] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6926.317161] LustreError: 239391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6926.353801] LustreError: 239391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6930.046946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.231648] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6930.271252] Lustre: 239379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.284438] Lustre: Permanently deactivating lustre-MDT0001 [ 6930.304229] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6930.727616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6932.755134] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6932.761184] LustreError: 239422:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6932.780417] LustreError: 239422:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6932.804637] LustreError: 239422:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6935.725996] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6943.147953] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6949.395062] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6956.830780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.433697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.516204] LustreError: 241231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.544168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6963.639504] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6966.781072] LustreError: 241233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.046755] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6971.882200] LustreError: 241231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.973706] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6978.959470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6979.383872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6984.712078] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6984.712933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6985.817939] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6993.453355] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6996.624909] Lustre: Permanently reactivating lustre-MDT0001 [ 6996.643263] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6996.648242] Lustre: Skipped 2 previous similar messages [ 6999.023760] Lustre: lustre-MDT0001: Client dbf6dc30-f442-4d71-bac5-3cbbf696c844 (at 192.168.201.18@tcp) reconnecting [ 7005.640886] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7007.120492] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7012.444624] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7012.610950] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7018.090480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7018.386509] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7021.928433] Lustre: Permanently deactivating lustre-MDT0001 [ 7021.969498] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7021.972874] Lustre: Skipped 2 previous similar messages [ 7031.502683] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7043.058964] Lustre: server umount lustre-MDT0000 complete [ 7043.553518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7043.557709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.586030] LustreError: 239391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.607334] LustreError: 239391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7046.124307] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.149042] Lustre: Skipped 1 previous similar message [ 7047.426039] LustreError: 239371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484359 with bad export cookie 17171432129437242380 [ 7047.430143] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.434142] LustreError: 239371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7047.786242] Lustre: server umount lustre-MDT0001 complete [ 7052.953234] Lustre: server umount lustre-OST0000 complete [ 7063.719210] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:59:35 (1789484375) [ 7077.875718] Lustre: server umount lustre-OST0001 complete [ 7087.620870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7099.727602] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7140.482551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.724971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7140.738411] Lustre: Skipped 2 previous similar messages [ 7140.765367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7140.836931] Lustre: lustre-MDT0000: new disk, initializing [ 7140.984231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.018299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7146.006236] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7155.843856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.031671] Lustre: 247701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7156.124804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7156.137912] Lustre: Skipped 1 previous similar message [ 7156.252925] Lustre: lustre-MDT0001: new disk, initializing [ 7156.390856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7156.451308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7156.463473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7161.175913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7169.944524] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7175.873706] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7182.808264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7183.059403] Lustre: lustre-OST0000: new disk, initializing [ 7183.062169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7183.065735] Lustre: 249683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7183.134407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7189.399904] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7190.073589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7190.081395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7190.230278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7198.074772] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7200.980379] LustreError: 247711:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7210.683492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.895383] Lustre: lustre-OST0001: new disk, initializing [ 7210.899957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7210.908259] Lustre: 250795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7210.983028] LustreError: 247711:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7211.028666] LustreError: 247709:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7218.409688] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7218.897812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7218.906739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7218.956758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7221.031143] LustreError: 247709:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7228.362867] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7230.061663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7234.528758] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7234.532472] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.535373] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7234.562589] Lustre: Skipped 2 previous similar messages [ 7239.656767] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7239.660157] Lustre: Skipped 2 previous similar messages [ 7244.778212] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7246.303147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7246.490399] Lustre: server umount lustre-OST0001 complete [ 7249.891462] LustreError: 249665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.917200] LustreError: 249665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7265.262383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.281033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7265.291389] Lustre: Skipped 1 previous similar message [ 7265.303386] Lustre: Skipped 2 previous similar messages [ 7270.383075] LustreError: 251768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.409438] LustreError: 251768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7270.637951] Lustre: server umount lustre-OST0000 complete [ 7279.584396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7279.595297] LustreError: Skipped 1 previous similar message [ 7279.599141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.618101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7290.335159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7290.620624] Lustre: server umount lustre-MDT0000 complete [ 7294.274365] LustreError: 248508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484606 with bad export cookie 17171432129437245194 [ 7294.276315] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.286503] LustreError: 248508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7300.720949] Lustre: server umount lustre-MDT0001 complete [ 7311.776691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7314.302242] Key type lgssc unregistered [ 7314.560088] LNet: 252837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7314.566150] LNetError: 252837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7314.582316] LNet: Removed LNI 192.168.201.118@tcp [ 7315.540640] Key type .llcrypt unregistered [ 7315.544977] Key type ._llcrypt unregistered [ 7354.247721] Key type ._llcrypt registered [ 7354.250812] Key type .llcrypt registered [ 7354.355220] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7355.444269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7355.486491] alg: No test for adler32 (adler32-zlib) [ 7356.567724] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 7356.815614] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7358.514409] Key type lgssc registered [ 7359.791547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.210258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7368.226360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.571181] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7369.606970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7369.728239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.422416] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7382.953923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.102961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7383.149830] Lustre: 255048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7383.440756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7388.224763] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7395.406519] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7400.603103] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.656656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.879081] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7408.093389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.493608] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7417.977171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7422.820924] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7479.271494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7479.288269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.297780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7479.776517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7483.431611] Lustre: server umount lustre-OST0000 complete [ 7487.341365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7501.987459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.546354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.185469] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7519.926166] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7519.949363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7519.961646] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7519.969338] Lustre: Skipped 1 previous similar message [ 7520.248252] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7566.306809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.309559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7566.326442] Lustre: Skipped 1 previous similar message [ 7566.345513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.345523] Lustre: Skipped 1 previous similar message [ 7568.605165] Lustre: server umount lustre-OST0000 complete [ 7572.961418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7572.977708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.009811] Lustre: Skipped 1 previous similar message [ 7573.014906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7573.023622] Lustre: Skipped 1 previous similar message [ 7576.544612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.548522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7576.579494] Lustre: Skipped 1 previous similar message [ 7579.325782] Lustre: server umount lustre-MDT0000 complete [ 7583.469900] LustreError: 255039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484895 with bad export cookie 9065846407933355401 [ 7583.471609] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7583.479884] LustreError: 255039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7583.825674] Lustre: server umount lustre-MDT0001 complete [ 7597.356372] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7600.760719] Key type lgssc unregistered [ 7601.133830] LNet: 260479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7601.139708] LNetError: 260479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7601.159859] LNet: Removed LNI 192.168.201.118@tcp [ 7602.238102] Key type .llcrypt unregistered [ 7602.239827] Key type ._llcrypt unregistered [ 7629.249571] Key type ._llcrypt registered [ 7629.252984] Key type .llcrypt registered [ 7629.376521] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7630.815294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7630.843206] alg: No test for adler32 (adler32-zlib) [ 7631.838301] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 7632.029364] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7633.689176] Key type lgssc registered [ 7634.935645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7644.388978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7644.433420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.989925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7651.246528] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7656.425142] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.440853] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7660.809259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.148949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7666.414853] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7674.463810] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7680.519771] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7687.722482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7688.295752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7691.399168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7695.891890] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7707.875603] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.949746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.953502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.990194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7743.022300] Lustre: Skipped 1 previous similar message [ 7744.944906] Lustre: server umount lustre-OST0000 complete [ 7753.192047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7753.193418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.208159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7753.208168] Lustre: Skipped 1 previous similar message [ 7753.260759] Lustre: Skipped 1 previous similar message [ 7756.208576] Lustre: server umount lustre-MDT0000 complete [ 7758.304148] LustreError: 262855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.332686] LustreError: 262855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7760.035639] LustreError: 261879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485072 with bad export cookie 2522845108641816670 [ 7760.061259] LustreError: 261879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7760.066889] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.669704] Lustre: server umount lustre-MDT0001 complete [ 7776.422993] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7779.953291] Key type lgssc unregistered [ 7780.315806] LNet: 266211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7780.325771] LNetError: 266211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7780.347059] LNet: Removed LNI 192.168.201.118@tcp [ 7781.553059] Key type .llcrypt unregistered [ 7781.556946] Key type ._llcrypt unregistered [ 7804.732676] Key type ._llcrypt registered [ 7804.735175] Key type .llcrypt registered [ 7804.833378] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7806.132063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.239159] alg: No test for adler32 (adler32-zlib) [ 7807.458733] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 7808.037561] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7809.799284] Key type lgssc registered [ 7811.167939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7821.147860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7821.164184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.842161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.985879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7832.039245] LustreError: 267436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.067996] LustreError: 267436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7837.154603] LustreError: 267435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.424023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.955726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.831862] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7850.510026] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7856.190600] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7863.079084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.394364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.574322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7870.793235] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7879.632688] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7895.530294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7895.533791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.629740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7897.146833] Lustre: server umount lustre-OST0000 complete [ 7903.716985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7903.727977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7903.736566] Lustre: Skipped 1 previous similar message [ 7903.754888] Lustre: Skipped 1 previous similar message [ 7908.854260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7908.859796] Lustre: Skipped 1 previous similar message [ 7913.122038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7913.249848] Lustre: server umount lustre-MDT0000 complete [ 7918.561118] LustreError: 268438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.591757] LustreError: 268438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7919.090771] LustreError: 267416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485231 with bad export cookie 11088638929656209299 [ 7919.114551] LustreError: 267416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7919.118957] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7919.728598] Lustre: server umount lustre-MDT0001 complete [ 7937.837211] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7942.396610] Key type lgssc unregistered [ 7942.790856] LNet: 271118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7942.805621] LNetError: 271118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7943.848951] LNet: Removed LNI 192.168.201.118@tcp [ 7945.300266] Key type .llcrypt unregistered [ 7945.303843] Key type ._llcrypt unregistered [ 7978.446443] Key type ._llcrypt registered [ 7978.450609] Key type .llcrypt registered [ 7978.695324] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7981.064646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7981.172577] alg: No test for adler32 (adler32-zlib) [ 7982.702677] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 7983.133597] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7984.823275] Key type lgssc registered [ 7987.180051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7997.336386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7997.382230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.962883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.941380] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8009.195318] LustreError: 272538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8009.242078] LustreError: 272538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8014.310351] LustreError: 272537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.916690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.625961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8026.031938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8037.764887] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8044.509833] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8053.297610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.622708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8060.226695] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8069.868628] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8070.015899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8098.786963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8098.799993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.833791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.852660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.861927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.864490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.893491] Lustre: Skipped 1 previous similar message [ 8104.128778] Lustre: server umount lustre-OST0000 complete [ 8110.560754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8110.572193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8110.606220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8115.936256] Lustre: server umount lustre-MDT0000 complete [ 8116.199530] LustreError: 272533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.216396] LustreError: 272533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8120.668262] LustreError: 272518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485432 with bad export cookie 15251737117598039996 [ 8120.698405] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8121.086129] Lustre: server umount lustre-MDT0001 complete [ 8135.951667] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8140.275613] Key type lgssc unregistered [ 8140.781779] LNet: 276852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8140.795353] LNetError: 276852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8140.818277] LNet: Removed LNI 192.168.201.118@tcp [ 8142.235815] Key type .llcrypt unregistered [ 8142.238744] Key type ._llcrypt unregistered [ 8163.760890] Key type ._llcrypt registered [ 8163.769839] Key type .llcrypt registered [ 8163.926976] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8165.357996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8165.584117] alg: No test for adler32 (adler32-zlib) [ 8166.637626] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 8166.877408] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8168.567750] Key type lgssc registered [ 8170.181470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8180.891635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8180.916449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.415568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.719609] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8191.990295] LustreError: 278090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8192.007738] LustreError: 278090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8197.094144] LustreError: 278092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.825515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8200.177097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8206.865459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8216.459190] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8223.563774] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8233.070847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8233.486041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.254374] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8249.355456] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8251.887852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8267.234025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.238182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8267.250303] Lustre: Skipped 1 previous similar message [ 8267.274929] Lustre: Skipped 1 previous similar message [ 8272.363941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.379272] Lustre: Skipped 1 previous similar message [ 8273.849569] Lustre: server umount lustre-OST0000 complete [ 8282.081166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8282.091377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8282.115824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8285.008483] Lustre: server umount lustre-MDT0000 complete [ 8287.717678] LustreError: 278079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.757369] LustreError: 278079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8289.336452] LustreError: 278057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485601 with bad export cookie 82011520012211656 [ 8289.339609] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8289.372116] LustreError: 278057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8290.033750] Lustre: server umount lustre-MDT0001 complete [ 8305.610962] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8310.628845] Key type lgssc unregistered [ 8310.994372] LNet: 281798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8310.999609] LNetError: 281798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8311.014984] LNet: Removed LNI 192.168.201.118@tcp [ 8312.059550] Key type .llcrypt unregistered [ 8312.064611] Key type ._llcrypt unregistered [ 8331.501664] Key type ._llcrypt registered [ 8331.506188] Key type .llcrypt registered [ 8331.665522] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8353.418179] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8373.233575] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8374.659034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8374.692383] alg: No test for adler32 (adler32-zlib) [ 8375.761272] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 8376.064536] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8377.896297] Key type lgssc registered [ 8379.630216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8427.333476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8427.384804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.874418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8428.910672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8429.004413] Lustre: lustre-MDT0000: new disk, initializing [ 8429.132332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.160353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8435.907844] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8449.914236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.133852] Lustre: 285371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8450.229726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8450.238110] Lustre: Skipped 1 previous similar message [ 8450.346747] Lustre: lustre-MDT0001: new disk, initializing [ 8450.532202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8450.588591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8450.605037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8457.188071] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8469.438042] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8477.961719] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8486.922428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.337266] Lustre: lustre-OST0000: new disk, initializing [ 8487.343487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8487.356204] Lustre: 287342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8487.536852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.764147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8494.781086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8495.002257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8496.368764] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8507.113165] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8513.916151] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8514.235524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8521.563100] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8521.878621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8525.796068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8525.814179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8525.844970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8525.860079] Lustre: Skipped 1 previous similar message [ 8530.924618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8530.939685] Lustre: Skipped 1 previous similar message [ 8536.033037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.054786] Lustre: Skipped 1 previous similar message [ 8539.103128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8539.594789] Lustre: server umount lustre-OST0000 complete [ 8546.282912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.289570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8546.323822] Lustre: Skipped 2 previous similar messages [ 8546.351594] Lustre: Skipped 1 previous similar message [ 8550.965271] Lustre: server umount lustre-MDT0000 complete [ 8556.433666] LustreError: 285363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485868 with bad export cookie 18428315827856906737 [ 8556.436940] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.452598] LustreError: 285363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8557.217197] Lustre: server umount lustre-MDT0001 complete [ 8568.057603] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:24:39 (1789485879) [ 8576.932785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8577.551751] LustreError: 289757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.578360] LustreError: 289757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8577.629763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.634619] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8583.727304] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8587.745792] LustreError: 289757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8592.878460] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.174370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8596.416709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8603.348718] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8614.175279] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.048794] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8628.277211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.915158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.817567] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8645.553839] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8661.984752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8661.998759] LustreError: Skipped 1 previous similar message [ 8662.007698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.030225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8663.013738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8667.182732] Lustre: server umount lustre-OST0000 complete [ 8671.965000] Lustre: server umount lustre-MDT0000 complete [ 8673.249564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8673.256140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8673.261585] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8673.272214] Lustre: Skipped 2 previous similar messages [ 8673.310291] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8676.229823] LustreError: 289737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485988 with bad export cookie 18428315827856907619 [ 8676.239836] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8676.241663] LustreError: 289737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8676.555242] Lustre: server umount lustre-MDT0001 complete [ 8689.615893] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8693.476988] Key type lgssc unregistered [ 8693.906630] LNet: 293320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8693.917674] LNetError: 293320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8693.936176] LNet: Removed LNI 192.168.201.118@tcp [ 8695.135418] Key type .llcrypt unregistered [ 8695.138569] Key type ._llcrypt unregistered [ 8727.413435] Key type ._llcrypt registered [ 8727.415799] Key type .llcrypt registered [ 8727.644905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8751.946736] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8753.604323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8753.652044] alg: No test for adler32 (adler32-zlib) [ 8754.714862] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 8755.096566] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8756.823181] Key type lgssc registered [ 8758.320944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8810.043195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8810.082365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.582805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8811.633571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8811.727320] Lustre: lustre-MDT0000: new disk, initializing [ 8811.876110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8811.925307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8817.036304] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8828.516734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.686876] Lustre: 297152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8828.753623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8828.759885] Lustre: Skipped 1 previous similar message [ 8828.872631] Lustre: lustre-MDT0001: new disk, initializing [ 8829.041746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8829.103291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8829.123620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8834.324870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8844.131166] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8850.150750] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8857.626255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8858.014584] Lustre: lustre-OST2710: new disk, initializing [ 8858.018441] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8858.028841] Lustre: 299121:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8858.118450] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8864.874601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8867.927713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8867.952587] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8868.096180] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8875.470524] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8883.342225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8883.502594] Lustre: lustre-OST03e8: new disk, initializing [ 8883.506251] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8883.516259] Lustre: 300187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8883.680266] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8890.729113] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8891.998054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8892.008906] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8892.138456] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8898.166854] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8905.637263] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8905.914141] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8910.463938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8910.687436] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8915.905568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8916.130436] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8920.692984] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8920.879344] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8926.176330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8926.184085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8926.211886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8927.928139] Lustre: server umount lustre-MDT0000 complete [ 8928.236968] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.256482] Lustre: Skipped 2 previous similar messages [ 8931.685807] LustreError: 297144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486243 with bad export cookie 7386651053502080573 [ 8931.687661] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8931.703860] LustreError: 297144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8932.347871] Lustre: server umount lustre-MDT0001 complete [ 8946.929499] Lustre: server umount lustre-OST2710 complete [ 8949.730360] Lustre: 294952:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789486245/real 1789486245] req@000000003968976c x1876412142259456/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789486261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8949.764036] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.171751] Lustre: server umount lustre-OST03e8 complete [ 8970.017826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8981.401950] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9017.109445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.281438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9017.298223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9017.371945] Lustre: lustre-MDT0000: new disk, initializing [ 9017.492676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.512675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9021.795858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9033.660844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9033.799649] Lustre: 305437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9033.863853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9033.868393] Lustre: Skipped 1 previous similar message [ 9033.956610] Lustre: lustre-MDT0001: new disk, initializing [ 9034.054639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9034.116198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9034.145279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9038.794922] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9048.698947] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9054.514409] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9062.025118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9062.281049] Lustre: lustre-OST0000: new disk, initializing [ 9062.288689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9062.294742] Lustre: 307405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9062.401289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9068.140151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9068.160471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9068.261741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9068.313984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9077.768494] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9082.822526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9083.093566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9089.071426] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9089.340350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9094.112824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.127906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.145239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.167263] Lustre: Skipped 1 previous similar message [ 9099.238187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9099.246085] Lustre: Skipped 2 previous similar messages [ 9104.356257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9104.359320] Lustre: Skipped 1 previous similar message [ 9105.887129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9106.047048] Lustre: server umount lustre-OST0000 complete [ 9111.008068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9111.013788] LustreError: Skipped 1 previous similar message [ 9111.015120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9111.032731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9119.713092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9119.723036] Lustre: Skipped 3 previous similar messages [ 9124.324093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9124.602920] Lustre: server umount lustre-MDT0000 complete [ 9124.834689] LustreError: 305442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.871283] LustreError: 305442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9128.044535] LustreError: 305429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486440 with bad export cookie 7386651053502082253 [ 9128.045464] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9128.057917] LustreError: 305429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9128.499478] Lustre: server umount lustre-MDT0001 complete [ 9137.764706] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:34:08 (1789486448) [ 9139.699603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9142.164336] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:34:13 (1789486453) [ 9147.845593] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9174.357445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9174.582962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9174.598817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9174.732569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.547321] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9187.642156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.835973] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9187.920913] Lustre: 311267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9193.124386] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9214.570682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9214.786139] Lustre: Denying initial registration attempt from nid 192.168.201.118@tcp, specified as failover [ 9214.796106] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9214.801792] LustreError: 312720:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9214.805501] LustreError: 312720:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9214.807741] LustreError: 312720:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9214.864842] Lustre: server umount lustre-OST0000 complete [ 9214.867784] LustreError: 312720:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9221.601115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.614571] Lustre: Skipped 1 previous similar message [ 9221.620468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9226.730907] LustreError: 311279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9226.963121] Lustre: server umount lustre-MDT0000 complete [ 9231.277091] LustreError: 311260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486543 with bad export cookie 7386651053502083135 [ 9231.293893] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9231.294580] LustreError: 311260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9231.908513] Lustre: server umount lustre-MDT0001 complete [ 9244.899677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9247.595775] Key type lgssc unregistered [ 9247.914897] LNet: 313843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9247.920237] LNetError: 313843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9248.939400] LNet: Removed LNI 192.168.201.118@tcp [ 9249.776855] Key type .llcrypt unregistered [ 9249.778463] Key type ._llcrypt unregistered [ 9262.282689] Key type ._llcrypt registered [ 9262.287496] Key type .llcrypt registered [ 9262.455986] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9264.031958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9264.090670] alg: No test for adler32 (adler32-zlib) [ 9265.270386] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 9265.650120] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9267.455519] Key type lgssc registered [ 9269.449706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9316.631108] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9325.368572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9325.397755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.714092] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9326.735771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9326.890207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.118722] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9341.200503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.313350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9341.347906] Lustre: 316609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9341.669353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9346.490516] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9354.095621] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9359.496539] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9379.841067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9380.131217] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9380.350302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.037205] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9396.919905] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.314667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9401.318080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.328289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9404.778534] Lustre: server umount lustre-OST0000 complete [ 9408.483830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9408.496874] LustreError: Skipped 1 previous similar message [ 9408.504103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9408.515724] Lustre: Skipped 1 previous similar message [ 9408.521367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9408.529210] Lustre: Skipped 1 previous similar message [ 9411.552305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9411.553539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.797280] Lustre: server umount lustre-MDT0000 complete [ 9418.521922] LustreError: 316600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486730 with bad export cookie 12441840079768000263 [ 9418.526819] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9418.532238] LustreError: 316600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9419.184878] Lustre: server umount lustre-MDT0001 complete [ 9429.581799] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9432.531278] Key type lgssc unregistered [ 9432.781127] LNet: 320186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9432.784436] LNetError: 320186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9433.831735] LNet: Removed LNI 192.168.201.118@tcp [ 9434.579692] Key type .llcrypt unregistered [ 9434.582184] Key type ._llcrypt unregistered [ 9461.436442] Key type ._llcrypt registered [ 9461.438906] Key type .llcrypt registered [ 9461.591203] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9463.087701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9463.203387] alg: No test for adler32 (adler32-zlib) [ 9464.369942] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 9464.692654] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9466.408387] Key type lgssc registered [ 9467.753923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9475.332968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9475.356907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.986104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9482.631125] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9487.329929] LustreError: 321601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.362458] LustreError: 321601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9491.270398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.532113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9495.586340] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9502.960256] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9509.391265] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9517.931918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9518.310141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9523.704531] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9530.790326] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9547.246577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.252969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9547.259560] Lustre: Skipped 1 previous similar message [ 9547.269304] Lustre: Skipped 2 previous similar messages [ 9548.910925] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 9551.477354] Lustre: server umount lustre-MDT0000 complete [ 9552.353614] LustreError: 321601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.371957] LustreError: 321601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9553.999734] LustreError: 323967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.833241] LustreError: 321583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486867 with bad export cookie 5894936822128568993 [ 9554.843375] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.473121] LustreError: 321595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9557.473598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.505115] Lustre: Skipped 1 previous similar message [ 9557.523678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9564.233057] LustreError: 321597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.233078] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 9564.253650] LustreError: 321597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9564.277270] Lustre: Skipped 2 previous similar messages [ 9569.247211] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9569.553486] Lustre: server umount lustre-MDT0001 complete [ 9576.899359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9585.486680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.479782] LustreError: 320911:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e2f38677 x1876412885676672/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9601.022830] LustreError: 323433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9601.054037] LustreError: 323433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9601.231706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9601.305288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9606.539729] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9615.533853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.086085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9620.213740] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9621.571717] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9630.198614] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.081697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9638.357494] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9639.764254] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9646.054203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9646.061415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9646.067417] Lustre: Skipped 1 previous similar message [ 9646.080577] Lustre: Skipped 3 previous similar messages [ 9650.734169] Lustre: server umount lustre-OST0000 complete [ 9656.289873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.302241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9656.307119] Lustre: Skipped 1 previous similar message [ 9660.530980] Lustre: server umount lustre-MDT0000 complete [ 9661.408408] LustreError: 325223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.431373] LustreError: 325223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9664.167442] LustreError: 321581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486976 with bad export cookie 5894936822128576833 [ 9664.168579] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.175925] LustreError: 321581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9664.547087] Lustre: server umount lustre-MDT0001 complete [ 9675.424815] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9678.919550] Key type lgssc unregistered [ 9679.206928] LNet: 327893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9679.216964] LNetError: 327893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9679.229601] LNet: Removed LNI 192.168.201.118@tcp [ 9680.511662] Key type .llcrypt unregistered [ 9680.514570] Key type ._llcrypt unregistered [ 9711.085437] Key type ._llcrypt registered [ 9711.087118] Key type .llcrypt registered [ 9711.151704] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9712.085301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9712.259708] alg: No test for adler32 (adler32-zlib) [ 9713.581187] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [ 9714.039042] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9715.793697] Key type lgssc registered [ 9717.187962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9724.693122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9724.724495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.334976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.937486] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9736.673392] LustreError: 329313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.685876] LustreError: 329313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9739.160538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.556584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9744.525277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9752.618380] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9758.700994] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.874948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.883637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9763.895249] Lustre: Skipped 1 previous similar message [ 9768.113979] Lustre: server umount lustre-MDT0000 complete [ 9768.929204] LustreError: 330041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9768.961529] LustreError: 330041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9772.172239] LustreError: 329293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487084 with bad export cookie 3869888361824927073 [ 9772.182134] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.183927] LustreError: 329293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9772.410296] Lustre: server umount lustre-MDT0001 complete [ 9786.468831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.705652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9786.726420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9786.862477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.112302] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9799.166992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.273561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9799.303535] Lustre: 331900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9799.556195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9804.470550] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9812.446733] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9818.420724] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9828.014550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9828.273324] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9828.294569] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9828.301826] LustreError: 333784:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9828.311710] LustreError: 333784:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9828.318078] LustreError: 333784:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9828.433724] Lustre: server umount lustre-OST0000 complete [ 9828.435801] LustreError: 333784:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9838.892021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9839.146889] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9839.362214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9846.031814] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9848.963497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9853.445283] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9861.604630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9861.773335] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9861.912687] Lustre: lustre-OST0001: new disk, initializing [ 9861.918150] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9861.925207] Lustre: 335326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9862.042913] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9868.582568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9870.990833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9871.005418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9871.214710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9877.093211] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9881.572292] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9881.585888] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.586892] LustreError: Skipped 1 previous similar message [ 9881.592058] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9881.609812] Lustre: Skipped 1 previous similar message [ 9881.628650] Lustre: Skipped 2 previous similar messages [ 9886.691341] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9886.705978] Lustre: Skipped 1 previous similar message [ 9891.815716] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9893.856943] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9893.958480] Lustre: server umount lustre-OST0001 complete [ 9896.937340] LustreError: 334252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.970766] LustreError: 334252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9900.001833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9900.019502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.039760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9900.043694] Lustre: Skipped 1 previous similar message [ 9902.077459] LustreError: 334251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.094756] LustreError: 334251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9903.853674] Lustre: server umount lustre-OST0000 complete [ 9912.288615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9912.298265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9912.302665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.302674] Lustre: Skipped 3 previous similar messages [ 9912.319642] Lustre: Skipped 2 previous similar messages [ 9912.346602] LustreError: Skipped 1 previous similar message [ 9914.087569] Lustre: server umount lustre-MDT0000 complete [ 9917.412425] LustreError: 331911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.442728] LustreError: 331911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9918.351401] LustreError: 334266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487230 with bad export cookie 3869888361824927626 [ 9918.361673] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9918.365049] LustreError: 334266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9918.775954] Lustre: server umount lustre-MDT0001 complete [ 9930.935325] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9934.226745] Key type lgssc unregistered [ 9934.502206] LNet: 337291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9934.509501] LNetError: 337291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9935.590060] LNet: Removed LNI 192.168.201.118@tcp [ 9936.783991] Key type .llcrypt unregistered [ 9936.785740] Key type ._llcrypt unregistered [10012.528769] Key type ._llcrypt registered [10012.532943] Key type .llcrypt registered [10012.666933] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10031.456957] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10032.596403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10032.627978] alg: No test for adler32 (adler32-zlib) [10034.073225] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [10034.412627] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10036.111214] Key type lgssc registered [10037.697529] Lustre: Echo OBD driver; http://www.lustre.org/ [10078.802038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10078.945271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10080.513454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10080.563174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10080.762110] Lustre: lustre-MDT0000: new disk, initializing [10080.925747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10080.961211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10088.107467] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10100.152719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.317267] Lustre: 342246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10100.369947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10100.377197] Lustre: Skipped 1 previous similar message [10100.492625] Lustre: lustre-MDT0001: new disk, initializing [10100.675427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10100.762624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10100.775513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10106.268820] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10116.511199] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10122.217341] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10128.860249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.474629] Lustre: lustre-OST0000: new disk, initializing [10129.477503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10129.484412] Lustre: 344213:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10129.588387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.570784] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10137.799288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10137.824411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10138.091909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10148.196780] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10154.298484] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10154.589646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10159.856770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10160.120930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10163.683839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10163.702278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10163.717114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10168.452469] Lustre: server umount lustre-OST0000 complete [10173.922834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10173.941636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10173.949097] Lustre: Skipped 2 previous similar messages [10173.959345] Lustre: Skipped 1 previous similar message [10179.073667] Lustre: server umount lustre-MDT0000 complete [10182.919618] LustreError: 342237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487495 with bad export cookie 17101428649157152430 [10182.924353] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10182.940682] LustreError: 342237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10183.500378] Lustre: server umount lustre-MDT0001 complete [10192.347931] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:51:43 (1789487503) [10200.306120] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:51:52 (1789487512) [10209.169749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10209.791926] LustreError: 346968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.817237] LustreError: 346968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10209.891095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10215.205096] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10215.399396] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.526755] LustreError: 346968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10224.298719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.813118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10229.276144] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10236.512577] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10242.996344] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10251.371358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10251.700311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10258.095767] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10267.048778] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10276.057832] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10278.094951] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10279.662570] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10281.345794] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10283.032568] Lustre: Failing over lustre-MDT0000 [10283.433902] Lustre: server umount lustre-MDT0000 complete [10284.517364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10284.529662] LustreError: 347696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.539448] Lustre: Skipped 1 previous similar message [10284.569547] LustreError: 347696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10289.642873] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10289.668295] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10299.873344] Lustre: 340469:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789487596/real 1789487596] req@00000000ae86e546 x1876413483391104/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789487612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10299.879883] LustreError: 347696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10299.899855] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10299.919668] LustreError: 347696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10302.243463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.113504] LustreError: 340465:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fb80dc96 x1876413483395584/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10310.369301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10310.421431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10311.754442] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10315.054137] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10315.781644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10315.833434] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10315.885673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10325.389533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10327.343492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10331.620627] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10333.886076] Lustre: Failing over lustre-MDT0000 [10334.182269] Lustre: server umount lustre-MDT0000 complete [10336.225525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10336.235577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10336.245133] LustreError: 346963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.245143] LustreError: 346963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10336.298307] Lustre: Skipped 3 previous similar messages [10352.607194] Lustre: 340466:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789487648/real 1789487648] req@00000000a6f15c03 x1876413483411200/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789487664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10352.647212] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10352.985168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10361.827355] LustreError: 340465:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ea9b234a x1876413483416448/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10362.195304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10362.245341] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10363.985597] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10366.990280] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10367.480542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10367.489478] Lustre: Skipped 2 previous similar messages [10367.519600] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10367.580772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10377.013706] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10378.970058] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10387.938714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.950855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.970979] Lustre: Skipped 1 previous similar message [10387.995590] Lustre: Skipped 3 previous similar messages [10392.091510] Lustre: server umount lustre-MDT0000 complete [10393.061326] LustreError: 346968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.087761] LustreError: 346968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [10395.650220] LustreError: 349506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487707 with bad export cookie 17101428649157155146 [10395.652400] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10395.667834] LustreError: 349506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10396.125419] Lustre: server umount lustre-MDT0001 complete [10410.260462] Lustre: server umount lustre-OST0000 complete [10426.651639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.402672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10431.868960] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10439.785193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.441680] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10452.006491] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10457.123460] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10463.913235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10470.643822] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10472.376055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10479.568514] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10485.733148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10485.745153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.758996] Lustre: Skipped 1 previous similar message [10485.765369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10487.093874] Lustre: server umount lustre-OST0000 complete [10492.907185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10492.931341] Lustre: Skipped 1 previous similar message [10501.169230] Lustre: server umount lustre-MDT0000 complete [10502.114481] LustreError: 354038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.131596] LustreError: 354038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10504.698454] LustreError: 355888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487816 with bad export cookie 17101428649157155895 [10504.700812] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10504.717413] LustreError: 355888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10505.076480] Lustre: server umount lustre-MDT0001 complete [10519.412884] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10522.300279] Key type lgssc unregistered [10522.571887] LNet: 356845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10522.579504] LNetError: 356845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10522.591514] LNet: Removed LNI 192.168.201.118@tcp [10523.412224] Key type .llcrypt unregistered [10523.415450] Key type ._llcrypt unregistered [10551.603662] Key type ._llcrypt registered [10551.607366] Key type .llcrypt registered [10551.697092] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10552.697201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10552.848141] alg: No test for adler32 (adler32-zlib) [10553.863617] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [10554.089230] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10555.759375] Key type lgssc registered [10557.057520] Lustre: Echo OBD driver; http://www.lustre.org/ [10566.167784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10566.204721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.763394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10572.743095] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10576.866892] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10576.888515] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10581.278932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10581.719052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10586.085482] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10594.608979] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10601.006272] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10607.573126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10608.013545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10612.086658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10615.964395] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10623.839536] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10638.311284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10638.319737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10638.333584] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.394984] Lustre: server umount lustre-MDT0000 complete [10641.889671] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.893918] LustreError: 359093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.918977] LustreError: 359093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10642.215088] LustreError: 358347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487954 with bad export cookie 1274791186247270591 [10642.216322] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.221174] LustreError: 358347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10642.585409] Lustre: server umount lustre-MDT0001 complete [10657.071898] Lustre: server umount lustre-OST0000 complete [10672.609946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.018275] LustreError: 362100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10673.078985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10677.841805] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10678.243919] LustreError: 362101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.860766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10687.431090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10692.381344] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10700.272471] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10706.780898] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10714.388073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.899025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10718.003040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10721.247222] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10728.715840] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10742.753963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.764029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10742.781771] Lustre: Skipped 2 previous similar messages [10742.794985] Lustre: Skipped 2 previous similar messages [10747.036512] Lustre: server umount lustre-MDT0000 complete [10747.874263] LustreError: 362095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.904485] LustreError: 362095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10751.182804] LustreError: 362081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488063 with bad export cookie 1274791186247271704 [10751.189353] LustreError: 362081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10751.190124] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10751.524359] Lustre: server umount lustre-MDT0001 complete [10765.817260] Lustre: server umount lustre-OST0000 complete [10782.645323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.172768] LustreError: 366020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10783.253437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10788.474955] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10797.749247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.791838] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10811.021839] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10817.553972] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10825.834884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.179798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10826.182586] Lustre: Skipped 1 previous similar message [10828.251830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10832.310441] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10842.163993] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10848.736729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10848.738053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10848.742581] LustreError: Skipped 1 previous similar message [10848.749909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.760547] Lustre: Skipped 2 previous similar messages [10848.773923] Lustre: Skipped 1 previous similar message [10850.012656] Lustre: server umount lustre-OST0000 complete [10858.977367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10858.979933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10859.001320] Lustre: Skipped 1 previous similar message [10864.099634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.105077] Lustre: Skipped 2 previous similar messages [10869.215144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10869.221942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10869.227286] Lustre: Skipped 1 previous similar message [10869.475473] Lustre: server umount lustre-MDT0000 complete [10873.301891] LustreError: 366002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488185 with bad export cookie 1274791186247272719 [10873.309209] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10873.314914] LustreError: 366002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10873.601606] Lustre: server umount lustre-MDT0001 complete [10886.747582] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10890.359721] Key type lgssc unregistered [10890.645836] LNet: 369557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10890.654826] LNetError: 369557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10890.669924] LNet: Removed LNI 192.168.201.118@tcp [10891.596700] Key type .llcrypt unregistered [10891.603280] Key type ._llcrypt unregistered [10920.246566] Key type ._llcrypt registered [10920.255924] Key type .llcrypt registered [10920.469458] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10921.891967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10922.004601] alg: No test for adler32 (adler32-zlib) [10923.219919] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [10923.493239] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10925.199220] Key type lgssc registered [10926.489890] Lustre: Echo OBD driver; http://www.lustre.org/ [10934.844334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10934.889724] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10934.905525] LustreError: 370998:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10934.978596] LustreError: 370998:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10934.985876] LustreError: 370998:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10934.992917] LustreError: 370998:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10935.005903] LustreError: 370998:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10943.911062] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10943.921182] LustreError: 371397:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10943.954101] LustreError: 371397:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10943.965840] LustreError: 371397:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10943.983157] LustreError: 371397:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10943.993907] LustreError: 371397:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10965.366789] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10969.432782] Key type lgssc unregistered [10969.973775] LNet: 372403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10969.995206] LNetError: 372403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10970.012175] LNet: Removed LNI 192.168.201.118@tcp [10970.895416] Key type .llcrypt unregistered [10970.901224] Key type ._llcrypt unregistered [10991.554330] Key type ._llcrypt registered [10991.555809] Key type .llcrypt registered [10991.665845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11010.342698] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11012.233592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11012.658394] alg: No test for adler32 (adler32-zlib) [11014.134777] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [11014.599537] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11016.439286] Key type lgssc registered [11018.592702] Lustre: Echo OBD driver; http://www.lustre.org/ [11059.843084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11059.876385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.303818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11061.335986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11061.480747] Lustre: lustre-MDT0000: new disk, initializing [11061.617939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.657854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11067.485603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11079.116789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.300063] Lustre: 375577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11079.380966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11079.385753] Lustre: Skipped 1 previous similar message [11079.463117] Lustre: lustre-MDT0001: new disk, initializing [11079.560615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11079.598083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11079.616732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11085.077261] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11096.361145] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11102.979468] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11111.157974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11111.437664] Lustre: lustre-OST0000: new disk, initializing [11111.441702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11111.448612] Lustre: 377545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11111.561442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11117.693463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11117.705320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11117.823342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11118.699172] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11129.168586] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11134.332518] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11134.601884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11140.142603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11140.383818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11143.674548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11143.680889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.749646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11148.561893] Lustre: server umount lustre-OST0000 complete [11153.898258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11153.911134] Lustre: Skipped 1 previous similar message [11153.915644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11153.920187] Lustre: Skipped 1 previous similar message [11159.011039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11159.027949] Lustre: Skipped 1 previous similar message [11163.105801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11163.108600] Lustre: Skipped 1 previous similar message [11168.223882] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11168.225138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11168.240649] Lustre: Skipped 1 previous similar message [11168.436981] Lustre: server umount lustre-MDT0000 complete [11172.243434] LustreError: 375567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488484 with bad export cookie 12624710882318663886 [11172.247246] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11172.254329] LustreError: 375567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11172.603125] Lustre: server umount lustre-MDT0001 complete [11180.995894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11182.948325] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:08:14 (1789488494) [11189.658953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.365676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11195.361134] LustreError: 380008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.384158] LustreError: 380008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11195.389136] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11200.486337] LustreError: 380007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.530576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.888387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11210.157177] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11219.117455] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11226.240517] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11234.388612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.679532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11241.895718] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11250.921973] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11258.236557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11258.451833] Lustre: lustre-OST0001: new disk, initializing [11258.455348] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11258.462255] Lustre: 382847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11258.585798] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11265.220597] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11265.638441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11265.649048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11265.751436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11274.379598] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11281.377934] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11281.395382] LustreError: Skipped 1 previous similar message [11281.412297] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.418161] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11281.455041] Lustre: Skipped 2 previous similar messages [11284.980826] Lustre: server umount lustre-OST0001 complete [11286.507564] LustreError: 381831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.538170] LustreError: 381831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11291.618497] LustreError: 381829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.677338] LustreError: 381829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11296.778621] LustreError: 382620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.794935] LustreError: 382620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11306.978908] LustreError: 381831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.983602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11306.984809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11306.984815] Lustre: Skipped 2 previous similar messages [11307.013061] LustreError: 381831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11307.065589] Lustre: Skipped 1 previous similar message [11312.078503] Lustre: server umount lustre-OST0000 complete [11317.216970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.364542] Lustre: server umount lustre-MDT0000 complete [11327.458467] LustreError: 380992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.496490] LustreError: 380992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11327.860244] LustreError: 379989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488640 with bad export cookie 12624710882318664768 [11327.868853] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.872131] LustreError: 379989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11342.306510] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11342.682282] Lustre: server umount lustre-MDT0001 complete [11354.798154] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11358.164643] Key type lgssc unregistered [11358.487664] LNet: 384841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11358.495704] LNetError: 384841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11358.514247] LNet: Removed LNI 192.168.201.118@tcp [11359.504402] Key type .llcrypt unregistered [11359.507719] Key type ._llcrypt unregistered [11416.533188] Key type ._llcrypt registered [11416.537572] Key type .llcrypt registered [11416.649566] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11418.315441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11418.578413] alg: No test for adler32 (adler32-zlib) [11419.760599] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [11420.135914] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11421.887732] Key type lgssc registered [11423.479392] Lustre: Echo OBD driver; http://www.lustre.org/ [11432.650275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11432.682791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11433.945416] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11433.972432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11434.141613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11439.791774] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11449.347285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.468544] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11449.504276] Lustre: 387642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11449.853691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11454.915399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11463.298097] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11470.138658] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11475.431890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11475.439924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.454033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11476.966054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11476.966546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11482.082565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11482.086521] Lustre: Skipped 2 previous similar messages [11487.200026] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11487.204991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11487.224390] Lustre: Skipped 1 previous similar message [11487.387818] Lustre: server umount lustre-MDT0000 complete [11491.933434] LustreError: 387637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488804 with bad export cookie 1117660134570915017 [11491.939982] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11491.949111] LustreError: 387637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11492.419466] Lustre: server umount lustre-MDT0001 complete [11505.831777] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:13:36 (1789488816) [11513.098457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.807991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.953315] LustreError: 390155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11518.977814] LustreError: 390155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11519.187929] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11524.067715] LustreError: 390156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.190357] LustreError: 390155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.735437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.972823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11538.169239] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11546.883369] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11553.325416] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11561.558029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11561.866440] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11562.135300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11569.723704] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11578.802182] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11588.063895] Lustre: Permanently deactivating lustre-OST0000 [11588.102849] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11594.271659] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11597.016885] LustreError: 393097:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11597.025627] LustreError: 393097:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11597.030996] LustreError: 393097:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11601.163466] LustreError: 393145:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11601.167865] LustreError: 393145:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11609.778580] Lustre: server umount lustre-OST0000 complete [11613.003912] LustreError: 393395:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11613.013694] LustreError: 393395:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11615.715335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.731124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11615.739552] Lustre: Skipped 1 previous similar message [11625.031791] Lustre: server umount lustre-MDT0000 complete [11625.954122] LustreError: 391595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.965667] LustreError: 391595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11629.964633] LustreError: 390136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488942 with bad export cookie 1117660134570915598 [11629.979051] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11640.595230] Lustre: server umount lustre-MDT0001 complete [11650.394837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11655.634651] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11659.493590] LustreError: 394557:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11666.779355] Lustre: 394896:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.118@tcp [11669.129848] Lustre: server umount MGS complete [11680.816046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.359618] LustreError: 395619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.388524] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11681.394400] Lustre: Skipped 1 previous similar message [11681.467973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11687.052590] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11692.009933] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.043033] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11697.362748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11697.850325] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11697.937171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11703.792524] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11712.738916] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11720.762876] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11723.318763] Lustre: Permanently reactivating lustre-OST0000 [11723.370901] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11723.382523] Lustre: Skipped 2 previous similar messages [11749.343153] LustreError: 397354:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11757.080878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11757.681249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.322093] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11778.939379] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11789.796000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11789.804825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11789.823341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11789.828761] Lustre: Skipped 2 previous similar messages [11789.864787] Lustre: Skipped 3 previous similar messages [11800.031211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11800.164933] Lustre: server umount lustre-OST0000 complete [11805.153480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.172458] Lustre: Skipped 1 previous similar message [11810.272161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11810.287131] Lustre: Skipped 6 previous similar messages [11810.959052] Lustre: server umount lustre-MDT0000 complete [11815.398890] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.434914] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11815.458461] LustreError: 395602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489127 with bad export cookie 1117660134570916648 [11815.467402] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11815.480602] LustreError: 395602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11815.950774] Lustre: server umount lustre-MDT0001 complete [11828.810139] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11832.671645] Key type lgssc unregistered [11833.012135] LNet: 399895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11833.025047] LNetError: 399895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11833.043625] LNet: Removed LNI 192.168.201.118@tcp [11833.648777] Key type .llcrypt unregistered [11833.651277] Key type ._llcrypt unregistered [11855.028718] Key type ._llcrypt registered [11855.041759] Key type .llcrypt registered [11855.185810] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11875.794936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11877.280174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11877.346412] alg: No test for adler32 (adler32-zlib) [11878.551888] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [11878.761878] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11880.431267] Key type lgssc registered [11882.471329] Lustre: Echo OBD driver; http://www.lustre.org/ [11923.287724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11923.333758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.566648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11924.591524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11924.648706] Lustre: lustre-MDT0000: new disk, initializing [11924.699752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11924.711913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11928.508653] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11938.599678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.746938] Lustre: 403052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11938.798216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11938.805289] Lustre: Skipped 1 previous similar message [11938.871647] Lustre: lustre-MDT0001: new disk, initializing [11938.952312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11938.993486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11939.003930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11946.067555] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11960.588587] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11967.891614] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11976.424872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11976.763170] Lustre: lustre-OST0000: new disk, initializing [11976.767329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11976.775131] Lustre: 405025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11976.898870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11985.091652] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11986.034695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11986.047698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11986.264637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11996.175366] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12001.507250] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12001.725570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12007.592660] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12007.828363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12012.008871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12012.009644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.021134] LustreError: Skipped 1 previous similar message [12012.038984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.049227] Lustre: Skipped 1 previous similar message [12015.015789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12015.022606] Lustre: Skipped 2 previous similar messages [12016.527245] Lustre: server umount lustre-OST0000 complete [12021.217278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12021.247605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12021.273384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12025.313739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.314403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12025.346746] Lustre: Skipped 1 previous similar message [12027.475616] Lustre: server umount lustre-MDT0000 complete [12031.534419] LustreError: 403045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489343 with bad export cookie 13095757862275559702 [12031.547426] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12031.551815] LustreError: 403045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12032.265465] Lustre: server umount lustre-MDT0001 complete [12043.146962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12044.864158] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:22:36 (1789489356) [12052.022777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12052.701115] LustreError: 407487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12052.711261] LustreError: 407487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12052.788755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12056.705691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12058.084211] LustreError: 407488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12062.182725] LustreError: 407487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.214528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12067.302047] LustreError: 407488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.795544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12074.746452] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12084.552663] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12091.708751] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12101.391335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12101.777064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12108.224698] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12118.186262] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12121.248352] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12125.913228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12137.957354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12137.959374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.960865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12137.973728] LustreError: Skipped 1 previous similar message [12137.979500] Lustre: Skipped 1 previous similar message [12140.742740] Lustre: server umount lustre-OST0000 complete [12148.197129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12148.204288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12148.207255] Lustre: Skipped 1 previous similar message [12148.222539] Lustre: Skipped 2 previous similar messages [12151.671143] Lustre: server umount lustre-MDT0000 complete [12153.314383] LustreError: 409941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12153.350596] LustreError: 409941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12155.728511] LustreError: 407468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489467 with bad export cookie 13095757862275560584 [12155.736313] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12155.746638] LustreError: 407468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12156.215268] Lustre: server umount lustre-MDT0001 complete [12169.254973] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12172.421492] Key type lgssc unregistered [12172.768466] LNet: 411218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.782376] LNetError: 411218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12173.861497] LNet: Removed LNI 192.168.201.118@tcp [12175.115778] Key type .llcrypt unregistered [12175.119037] Key type ._llcrypt unregistered [12198.093464] Key type ._llcrypt registered [12198.094892] Key type .llcrypt registered [12198.169901] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12198.472803] Key type .llcrypt unregistered [12198.475854] Key type ._llcrypt unregistered [12218.288683] Key type ._llcrypt registered [12218.291283] Key type .llcrypt registered [12218.399601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12219.849157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12219.861834] alg: No test for adler32 (adler32-zlib) [12220.938922] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12221.182464] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12222.815202] Key type lgssc registered [12223.938084] Lustre: Echo OBD driver; http://www.lustre.org/ [12233.251710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12233.298487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12235.256524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12242.773360] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12245.482527] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12245.522279] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12246.510039] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.625714] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.579779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.739441] LustreError: 413438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.312926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.410959] LustreError: 413433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.450764] LustreError: 413433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12261.493146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12266.812098] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12271.587983] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12271.614178] LustreError: 413439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12274.912983] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12282.543043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.091197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12288.524065] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12298.725902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12298.729326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12298.754692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12298.757678] Lustre: Skipped 1 previous similar message [12298.776478] Lustre: Skipped 1 previous similar message [12303.850691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12303.853778] Lustre: Skipped 1 previous similar message [12304.513958] Lustre: server umount lustre-OST0000 complete [12308.962675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12308.966752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12308.979059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12308.996899] Lustre: Skipped 1 previous similar message [12314.089257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12314.099185] Lustre: Skipped 2 previous similar messages [12315.347044] Lustre: server umount lustre-MDT0000 complete [12319.212468] LustreError: 413434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.235165] LustreError: 413434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12319.462675] LustreError: 413419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489631 with bad export cookie 6203231314592124068 [12319.465811] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12319.467208] LustreError: 413419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12319.622554] Lustre: server umount lustre-MDT0001 complete [12330.572927] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12333.012473] Key type lgssc unregistered [12333.455070] LNet: 416593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12333.466103] LNetError: 416593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12333.488466] LNet: Removed LNI 192.168.201.118@tcp [12334.588877] Key type .llcrypt unregistered [12334.595399] Key type ._llcrypt unregistered [12372.959731] Key type ._llcrypt registered [12372.967460] Key type .llcrypt registered [12373.209685] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12375.437821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12375.484187] alg: No test for adler32 (adler32-zlib) [12376.484423] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12376.885899] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12378.680439] Key type lgssc registered [12380.001368] Lustre: Echo OBD driver; http://www.lustre.org/ [12388.523469] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12388.542037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12404.063647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12409.183360] LustreError: 417986:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [12435.871267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12441.665862] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12449.239766] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12456.948169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.060448] LustreError: 417323:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d98b21d1 x1876415940135168/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12457.585423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12457.679327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12463.084757] LustreError: 418011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12463.120434] LustreError: 418011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12464.419723] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12468.200314] LustreError: 418012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.255106] LustreError: 419024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.274593] LustreError: 419024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12475.103745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12475.536015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12480.333376] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12489.102345] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12495.587382] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12506.084537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12506.096059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12506.121418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12506.610584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12506.641188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12510.498495] Lustre: server umount lustre-OST0000 complete [12516.322641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12516.333406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12516.357503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12520.900990] Lustre: server umount lustre-MDT0000 complete [12521.960390] LustreError: 419028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12521.977277] LustreError: 419028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12526.049241] LustreError: 419024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12526.054651] LustreError: 417992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489838 with bad export cookie 17613465771750409836 [12526.077263] LustreError: 419024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12526.086955] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12526.097448] LustreError: 417992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12526.811083] Lustre: server umount lustre-MDT0001 complete [12541.221977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12544.422812] Key type lgssc unregistered [12544.845694] LNet: 421562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12544.850882] LNetError: 421562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12544.863258] LNet: Removed LNI 192.168.201.118@tcp [12545.933278] Key type .llcrypt unregistered [12545.935562] Key type ._llcrypt unregistered [12573.266132] Key type ._llcrypt registered [12573.267536] Key type .llcrypt registered [12573.315842] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12574.432452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12574.476485] alg: No test for adler32 (adler32-zlib) [12575.543482] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12575.804103] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12577.513146] Key type lgssc registered [12578.812982] Lustre: Echo OBD driver; http://www.lustre.org/ [12589.519311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12589.546469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12591.083531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12597.706535] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12601.318202] LustreError: 422978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12601.350488] LustreError: 422978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12606.449760] LustreError: 422977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.502764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12607.944499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12611.975581] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12620.368656] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12627.359809] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12634.700294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12635.141025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12636.229325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12641.429226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12649.954395] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12658.305873] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [12658.662537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12658.696203] Lustre: Skipped 2 previous similar messages [12662.442777] Lustre: server umount lustre-MDT0000 complete [12663.370589] LustreError: 422974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.985795] LustreError: 422958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.18@tcp arrived at 1789489978 with bad export cookie 7606322206432367851 [12666.018665] LustreError: 422958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12667.606653] LustreError: 422960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489979 with bad export cookie 7606322206432366962 [12667.613679] LustreError: 422960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12667.615711] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12668.899808] LustreError: 422977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12668.901622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.909561] LustreError: 422977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12668.925287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12668.936698] Lustre: Skipped 3 previous similar messages [12672.929090] LustreError: 422978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12672.929678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12672.952810] LustreError: 422978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12674.021430] Lustre: server umount lustre-OST0000 complete [12681.937820] Lustre: server umount lustre-MDT0001 complete [12694.288776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12697.308857] Key type lgssc unregistered [12697.668801] LNet: 426665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12697.682515] LNetError: 426665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12697.700785] LNet: Removed LNI 192.168.201.118@tcp [12698.532916] Key type .llcrypt unregistered [12698.540263] Key type ._llcrypt unregistered [12730.840855] Key type ._llcrypt registered [12730.842321] Key type .llcrypt registered [12731.008384] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12732.875431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12732.963235] alg: No test for adler32 (adler32-zlib) [12734.239904] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12734.551806] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12736.335187] Key type lgssc registered [12737.961531] Lustre: Echo OBD driver; http://www.lustre.org/ [12748.636748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12748.661899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12750.305259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12756.502483] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12760.551751] LustreError: 428091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.579433] LustreError: 428091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12765.677734] LustreError: 428092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.514725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12767.027121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12772.283590] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12781.216517] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12788.011369] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12794.857656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12795.180525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12800.805989] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12808.313497] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12820.732808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12828.646348] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12828.655489] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12828.671950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12830.829452] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [12831.224274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12835.015801] Lustre: server umount lustre-MDT0001 complete [12835.946064] LustreError: 429074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.447732] LustreError: 428088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.473730] LustreError: 428088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12846.564209] LustreError: 428091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.566930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12846.609125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.636474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12846.643908] Lustre: Skipped 2 previous similar messages [12848.102446] Lustre: server umount lustre-OST0000 complete [12852.644847] Lustre: server umount lustre-MDT0000 complete [12868.045824] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12871.167565] Key type lgssc unregistered [12871.522512] LNet: 431802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12871.527736] LNetError: 431802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12871.536700] LNet: Removed LNI 192.168.201.118@tcp [12872.422376] Key type .llcrypt unregistered [12872.429435] Key type ._llcrypt unregistered [12904.531470] Key type ._llcrypt registered [12904.535635] Key type .llcrypt registered [12904.720758] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12924.517230] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12926.034136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12926.042790] alg: No test for adler32 (adler32-zlib) [12927.151535] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [12927.447029] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12929.167538] Key type lgssc registered [12931.107946] Lustre: Echo OBD driver; http://www.lustre.org/ [12977.550492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12997.160198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12997.188240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12998.648247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12998.700084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12998.847897] Lustre: lustre-MDT0000: new disk, initializing [12998.979541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12999.010955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13004.167929] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13020.236959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13020.698812] Lustre: lustre-OST0000: new disk, initializing [13020.705343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13020.712613] Lustre: Skipped 1 previous similar message [13020.717848] Lustre: 436839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13020.874345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13027.533938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13028.962876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13028.968474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13028.999433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13036.818125] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13056.480468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13056.706857] Lustre: 435820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13056.854699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13056.986384] Lustre: lustre-MDT0001: new disk, initializing [13057.183611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13057.248328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13057.261714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13062.082988] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13070.958409] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13071.254259] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13076.404307] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13076.638582] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13085.676354] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.709583] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13085.711957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13085.741376] Lustre: Skipped 1 previous similar message [13086.318906] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [13088.179581] Lustre: server umount lustre-MDT0001 complete [13095.906796] LustreError: 435827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13095.916749] LustreError: 435827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13101.047918] LustreError: 435832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13101.067269] LustreError: 435832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13106.151354] LustreError: 435826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13106.190198] LustreError: 435826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13110.761857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13110.786757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13110.799253] Lustre: Skipped 1 previous similar message [13110.838396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13111.277560] LustreError: 435831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13116.385059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13116.385874] LustreError: 435832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13116.390701] Lustre: Skipped 1 previous similar message [13120.993951] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13121.295543] Lustre: server umount lustre-OST0000 complete [13132.081065] Lustre: server umount lustre-MDT0000 complete [13149.949750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13154.321130] Key type lgssc unregistered [13154.753806] LNet: 440592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13154.769390] LNetError: 440592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13154.791966] LNet: Removed LNI 192.168.201.118@tcp [13156.097401] Key type .llcrypt unregistered [13156.102303] Key type ._llcrypt unregistered [13204.067660] Key type ._llcrypt registered [13204.068831] Key type .llcrypt registered [13204.178479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13204.485610] Key type .llcrypt unregistered [13204.487448] Key type ._llcrypt unregistered [13228.082101] Key type ._llcrypt registered [13228.086495] Key type .llcrypt registered [13228.174996] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13229.261475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13229.300969] alg: No test for adler32 (adler32-zlib) [13230.687483] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [13231.138642] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [13232.983803] Key type lgssc registered [13234.183775] Lustre: Echo OBD driver; http://www.lustre.org/ [13262.165092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13262.193568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13263.530148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13263.548364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13263.643388] Lustre: lustre-MDT0000: new disk, initializing [13263.751518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13263.765444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13269.339829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13279.002053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13279.160826] Lustre: 443975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13279.224213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13279.227277] Lustre: Skipped 1 previous similar message [13279.323510] Lustre: lustre-MDT0001: new disk, initializing [13279.439344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13279.503663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13279.525179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13283.891836] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13294.318552] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13299.822428] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13308.359796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13308.582436] Lustre: lustre-OST0000: new disk, initializing [13308.585831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13308.590202] Lustre: 445943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13308.676170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13315.130117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13315.141743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13315.207920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13315.508378] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13325.891157] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13336.052931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13336.059945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13336.098483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13336.125775] Lustre: Skipped 2 previous similar messages [13338.355135] Lustre: server umount lustre-MDT0000 complete [13343.371358] LustreError: 443968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490655 with bad export cookie 8381023661781030097 [13343.374925] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13343.377164] LustreError: 443968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13344.331778] Lustre: server umount lustre-MDT0001 complete [13359.224269] Lustre: server umount lustre-OST0000 complete [13370.794143] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:44:42 (1789490682) [13379.343750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.812415] LustreError: 447972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.832022] LustreError: 447972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13379.902923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13384.477903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13385.188312] LustreError: 447973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.309160] LustreError: 447972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.285703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13394.784738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13400.679308] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13408.636602] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13414.572591] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13435.516732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13435.947229] Lustre: Found index 0 for lustre-OST0000, updating log [13436.316523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13444.574989] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13451.640050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13453.841488] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13459.936671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13459.945499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13459.957526] Lustre: Skipped 2 previous similar messages [13459.968329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13461.048110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13461.066679] Lustre: Skipped 1 previous similar message [13467.106873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13467.121060] Lustre: Skipped 1 previous similar message [13473.252338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13473.261734] Lustre: Skipped 1 previous similar message [13474.271139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13474.563841] Lustre: server umount lustre-OST0000 complete [13492.192901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13492.206347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13492.220747] Lustre: Skipped 1 previous similar message [13492.230748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13494.431161] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [13494.754475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13497.279102] Lustre: server umount lustre-MDT0000 complete [13499.874517] LustreError: 447973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13499.897248] LustreError: 447973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13500.568262] LustreError: 447954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490812 with bad export cookie 8381023661781031483 [13500.572899] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13500.574695] LustreError: 447954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13501.123153] Lustre: server umount lustre-MDT0001 complete [13513.514268] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:47:04 (1789490824) [14140.364407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14140.901054] LustreError: 454389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14140.975694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14146.016240] LustreError: 454390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14146.396931] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14151.139107] LustreError: 454389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14154.868106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14155.468935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14159.813442] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14169.577821] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14178.213921] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14198.047654] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [14200.573051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14200.975780] Lustre: Found index 0 for lustre-OST0000, updating log [14201.418902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14208.894329] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14216.939720] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14220.392525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14220.405167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14220.411106] Lustre: Skipped 2 previous similar messages [14225.710333] Lustre: server umount lustre-OST0000 complete [14230.626342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14230.629354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14230.648918] Lustre: Skipped 1 previous similar message [14230.667645] Lustre: Skipped 1 previous similar message [14234.850616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14234.859427] Lustre: Skipped 7 previous similar messages [14235.826779] Lustre: server umount lustre-MDT0000 complete [14235.873432] LustreError: 454390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14235.884927] LustreError: 454390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14239.729816] LustreError: 454369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491551 with bad export cookie 8381023661781032575 [14239.745788] LustreError: 454369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14239.746286] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14240.355192] Lustre: server umount lustre-MDT0001 complete [14258.541839] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [14273.950746] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14284.805753] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14327.425871] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:00:38 (1789491638) [14346.033935] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [14349.852496] Key type lgssc unregistered [14350.260055] LNet: 461016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14350.268921] LNetError: 461016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14465.000417] LNet: Removed LNI 192.168.201.118@tcp [14466.668907] Key type .llcrypt unregistered [14466.671323] Key type ._llcrypt unregistered [14486.709139] Key type ._llcrypt registered [14486.713625] Key type .llcrypt registered [14486.935181] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14488.524719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14488.693408] alg: No test for adler32 (adler32-zlib) [14489.910574] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [14490.240526] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [14491.944413] Key type lgssc registered [14493.090506] Lustre: Echo OBD driver; http://www.lustre.org/ [14498.249959] LNet: Added LNI 192.168.201.118@tcp100 [8/256/0/180] [14499.100966] LNet: Added LNI 192.168.201.118@tcp101 [8/256/0/180] [14499.720741] LNet: Added LNI 192.168.201.118@tcp102 [8/256/0/180] [14500.866393] LNet: Added LNI 192.168.201.118@tcp103 [8/256/0/180] [14501.924684] LNet: Added LNI 192.168.201.118@tcp104 [8/256/0/180] [14502.928877] LNet: Added LNI 192.168.201.118@tcp105 [8/256/0/180] [14504.032441] LNet: Added LNI 192.168.201.118@tcp106 [8/256/0/180] [14505.272893] LNet: Added LNI 192.168.201.118@tcp107 [8/256/0/180] [14506.369620] LNet: Added LNI 192.168.201.118@tcp108 [8/256/0/180] [14507.360050] LNet: Added LNI 192.168.201.118@tcp109 [8/256/0/180] [14508.172108] LNet: Added LNI 192.168.201.118@tcp110 [8/256/0/180] [14509.144207] LNet: Added LNI 192.168.201.118@tcp111 [8/256/0/180] [14510.295218] LNet: Added LNI 192.168.201.118@tcp112 [8/256/0/180] [14511.417543] LNet: Added LNI 192.168.201.118@tcp113 [8/256/0/180] [14512.340514] LNet: Added LNI 192.168.201.118@tcp114 [8/256/0/180] [14513.235457] LNet: Added LNI 192.168.201.118@tcp115 [8/256/0/180] [14514.305925] LNet: Added LNI 192.168.201.118@tcp116 [8/256/0/180] [14515.085508] LNet: Added LNI 192.168.201.118@tcp117 [8/256/0/180] [14516.062073] LNet: Added LNI 192.168.201.118@tcp118 [8/256/0/180] [14517.167440] LNet: Added LNI 192.168.201.118@tcp119 [8/256/0/180] [14518.133206] LNet: Added LNI 192.168.201.118@tcp120 [8/256/0/180] [14519.073705] LNet: Added LNI 192.168.201.118@tcp121 [8/256/0/180] [14520.154421] LNet: Added LNI 192.168.201.118@tcp122 [8/256/0/180] [14521.152190] LNet: Added LNI 192.168.201.118@tcp123 [8/256/0/180] [14522.352090] LNet: Added LNI 192.168.201.118@tcp124 [8/256/0/180] [14523.761887] LNet: Added LNI 192.168.201.118@tcp125 [8/256/0/180] [14525.035137] LNet: Added LNI 192.168.201.118@tcp126 [8/256/0/180] [14526.065895] LNet: Added LNI 192.168.201.118@tcp127 [8/256/0/180] [14527.206551] LNet: Added LNI 192.168.201.118@tcp128 [8/256/0/180] [14528.319438] LNet: Added LNI 192.168.201.118@tcp129 [8/256/0/180] [14529.554591] LNet: Added LNI 192.168.201.118@tcp130 [8/256/0/180] [14530.624216] LNet: Added LNI 192.168.201.118@tcp131 [8/256/0/180] [14531.739741] LNet: Added LNI 192.168.201.118@tcp132 [8/256/0/180] [14532.805765] LNet: Added LNI 192.168.201.118@tcp133 [8/256/0/180] [14533.971608] LNet: Added LNI 192.168.201.118@tcp134 [8/256/0/180] [14535.078373] LNet: Added LNI 192.168.201.118@tcp135 [8/256/0/180] [14542.703811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14542.719912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14543.808729] Lustre: 464049:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.118@tcp: can't add NID 192.168.201.118@tcp130: rc = -75 [14543.896568] LNetError: 464079:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14543.913310] LNetError: Protocol error connecting to 192.168.201.118@tcp115 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14544.018828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14544.064517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14544.155268] Lustre: lustre-MDT0000: new disk, initializing [14544.234530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14544.263508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14544.628254] LNet: 6 local NIs in recovery (showing 6): 192.168.201.118@tcp115, 192.168.201.118@tcp125, 192.168.201.118@tcp127, 192.168.201.118@tcp129, 192.168.201.118@tcp124, 192.168.201.118@tcp128 [14544.634058] LNetError: 461506:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14544.653410] LNetError: 461506:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14544.667533] LNetError: Protocol error connecting to 192.168.201.118@tcp107 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14544.683085] LNetError: Skipped 10 previous similar messages [14546.727161] LNetError: 461505:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14546.751180] LNetError: 461505:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14546.769137] LNetError: Protocol error connecting to 192.168.201.118@tcp124 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14550.141173] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14550.826757] LNetError: 461507:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14550.853068] LNetError: Protocol error connecting to 192.168.201.118@tcp124 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14559.012779] LNetError: 461508:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14559.038965] LNetError: Protocol error connecting to 192.168.201.118@tcp124 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14564.324925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14564.623206] Lustre: 464067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14564.747874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14564.757672] Lustre: Skipped 1 previous similar message [14564.895033] Lustre: lustre-MDT0001: new disk, initializing [14564.965712] Lustre: 464868:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.118@tcp: can't add NID 192.168.201.118@tcp130: rc = -75 [14564.975992] Lustre: 464868:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14565.191945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14565.274457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14565.307583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14571.839391] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14572.569277] Lustre: 465150:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.118@tcp: can't add NID 192.168.201.118@tcp130: rc = -75 [14572.585215] Lustre: 465150:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14574.304631] LNetError: 464079:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14574.323187] LNetError: Protocol error connecting to 192.168.201.118@tcp122 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14585.221051] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14590.690019] LNetError: 464079:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14590.701500] LNetError: 464079:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14590.709347] LNetError: Protocol error connecting to 192.168.201.118@tcp107 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14590.726873] LNetError: Skipped 7 previous similar messages [14592.877824] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14600.818523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14601.160928] Lustre: lustre-OST0000: new disk, initializing [14601.164328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14601.170821] Lustre: 466043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14601.254390] Lustre: 466048:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.118@tcp: can't add NID 192.168.201.118@tcp130: rc = -75 [14601.265034] Lustre: 466048:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14601.334305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14604.130128] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp124 [14608.968311] Lustre: 466357:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.118@tcp: can't add NID 192.168.201.118@tcp130: rc = -75 [14608.978152] Lustre: 466357:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14609.064679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14609.084971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14609.302619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14609.455500] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14622.528594] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14624.548800] LNetError: 464082:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.118 [14624.588931] LNetError: 464082:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14624.612426] LNetError: Protocol error connecting to 192.168.201.118@tcp127 at host 192.168.201.118:988: is it running a compatible version of Lustre? [14624.623157] LNetError: Skipped 14 previous similar messages [14635.008304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14635.010245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14635.028018] LustreError: Skipped 1 previous similar message [14635.029321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14635.069288] Lustre: Skipped 1 previous similar message [14637.947212] Lustre: server umount lustre-OST0000 complete [14645.219869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14645.248450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14645.261714] Lustre: Skipped 2 previous similar messages [14649.633391] Lustre: server umount lustre-MDT0000 complete [14655.284367] LustreError: 464058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491967 with bad export cookie 10045854835633469450 [14655.286838] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14655.301437] LustreError: 464058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14656.004584] Lustre: server umount lustre-MDT0001 complete [14670.327521] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [14673.719986] Key type lgssc unregistered [14674.315897] LNet: 467959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14674.326385] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14674.334690] LNet: Removed LNI 192.168.201.118@tcp [14675.359404] LNet: Removed LNI 192.168.201.118@tcp100 [14675.374098] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14675.379658] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14675.384183] LNet: Removed LNI 192.168.201.118@tcp101 [14676.447207] LNet: Removed LNI 192.168.201.118@tcp102 [14676.451371] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14676.465377] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14676.480799] LNet: Removed LNI 192.168.201.118@tcp103 [14677.535916] LNet: Removed LNI 192.168.201.118@tcp104 [14678.559712] LNet: Removed LNI 192.168.201.118@tcp105 [14678.564287] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14678.582050] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14678.596568] LNet: Removed LNI 192.168.201.118@tcp106 [14679.647266] LNet: Removed LNI 192.168.201.118@tcp107 [14680.673787] LNet: Removed LNI 192.168.201.118@tcp108 [14681.695207] LNet: Removed LNI 192.168.201.118@tcp109 [14682.719634] LNet: Removed LNI 192.168.201.118@tcp110 [14682.721653] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14682.734356] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14682.748925] LNet: Removed LNI 192.168.201.118@tcp111 [14683.807245] LNet: Removed LNI 192.168.201.118@tcp112 [14684.833037] LNet: Removed LNI 192.168.201.118@tcp113 [14685.919927] LNet: Removed LNI 192.168.201.118@tcp114 [14686.943979] LNet: Removed LNI 192.168.201.118@tcp115 [14687.968290] LNet: Removed LNI 192.168.201.118@tcp116 [14688.991141] LNet: Removed LNI 192.168.201.118@tcp117 [14690.015165] LNet: Removed LNI 192.168.201.118@tcp118 [14691.039275] LNet: Removed LNI 192.168.201.118@tcp119 [14691.042524] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14691.049203] LNetError: 467959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14691.056145] LNet: Removed LNI 192.168.201.118@tcp120 [14692.063386] LNet: Removed LNI 192.168.201.118@tcp121 [14693.087234] LNet: Removed LNI 192.168.201.118@tcp122 [14694.111202] LNet: Removed LNI 192.168.201.118@tcp123 [14694.116682] LNet: Removed LNI 192.168.201.118@tcp124 [14695.199474] LNet: Removed LNI 192.168.201.118@tcp125 [14696.226127] LNet: Removed LNI 192.168.201.118@tcp126 [14697.311530] LNet: Removed LNI 192.168.201.118@tcp127 [14698.335228] LNet: Removed LNI 192.168.201.118@tcp128 [14699.359819] LNet: Removed LNI 192.168.201.118@tcp129 [14699.363884] LNet: Removed LNI 192.168.201.118@tcp130 [14699.371290] LNet: Removed LNI 192.168.201.118@tcp131 [14699.378251] LNet: Removed LNI 192.168.201.118@tcp132 [14699.394780] LNet: Removed LNI 192.168.201.118@tcp133 [14699.406454] LNet: Removed LNI 192.168.201.118@tcp134 [14699.423975] LNet: Removed LNI 192.168.201.118@tcp135 [14701.162769] Key type .llcrypt unregistered [14701.169367] Key type ._llcrypt unregistered [14734.527768] Key type ._llcrypt registered [14734.529414] Key type .llcrypt registered [14734.617267] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14751.813389] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14753.138749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14753.456273] alg: No test for adler32 (adler32-zlib) [14755.073817] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [14755.481451] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [14757.199619] Key type lgssc registered [14759.308938] Lustre: Echo OBD driver; http://www.lustre.org/ [14802.458768] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:08:33 (1789492113) [14826.152485] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [14829.453964] Key type lgssc unregistered [14829.966860] LNet: 472056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14829.993690] LNetError: 472056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14830.028708] LNet: Removed LNI 192.168.201.118@tcp [14830.770765] Key type .llcrypt unregistered [14830.772819] Key type ._llcrypt unregistered [14855.688755] Key type ._llcrypt registered [14855.691677] Key type .llcrypt registered [14855.779837] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14857.230698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14857.347649] alg: No test for adler32 (adler32-zlib) [14858.331191] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [14858.490164] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [14860.151351] Key type lgssc registered [14861.054833] Lustre: Echo OBD driver; http://www.lustre.org/ [14873.175426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14873.197537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14874.485176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14874.506519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14874.631914] Lustre: lustre-MDT0000: new disk, initializing [14874.754496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14874.772061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14880.111357] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14892.718949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14892.949327] Lustre: 473270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14893.052909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14893.058122] Lustre: Skipped 1 previous similar message [14893.133771] Lustre: lustre-MDT0001: new disk, initializing [14893.263517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14893.407313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14893.433598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14899.084735] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14908.397988] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14914.559384] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14923.733867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14924.245746] Lustre: lustre-OST0000: new disk, initializing [14924.251429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14924.262875] Lustre: 475242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14924.370384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14932.113371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14932.150726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14932.399167] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14932.450309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14944.175275] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14950.549707] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing lnet_if_list [14958.260903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14958.394797] LNet: Added LNI 192.168.201.118@tcp10 [8/256/0/180] [14964.721667] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14964.758372] LNet: Added LNI 192.168.201.118@tcp11 [8/256/0/180] [14969.863897] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14969.899213] LNet: Added LNI 192.168.201.118@tcp12 [8/256/0/180] [14976.268079] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14976.318768] LNet: Added LNI 192.168.201.118@tcp13 [8/256/0/180] [14981.299101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14981.353747] LNet: Added LNI 192.168.201.118@tcp14 [8/256/0/180] [14987.688066] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14987.702388] LNet: Added LNI 192.168.201.118@tcp15 [8/256/0/180] [14994.080451] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14994.112507] LNet: Added LNI 192.168.201.118@tcp16 [8/256/0/180] [15001.069213] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15001.144729] LNet: Added LNI 192.168.201.118@tcp17 [8/256/0/180] [15007.585984] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15007.642235] LNet: Added LNI 192.168.201.118@tcp18 [8/256/0/180] [15014.222614] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15014.262907] LNet: Added LNI 192.168.201.118@tcp19 [8/256/0/180] [15021.202856] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15021.264929] LNet: Added LNI 192.168.201.118@tcp20 [8/256/0/180] [15029.537209] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15029.575323] LNet: Added LNI 192.168.201.118@tcp21 [8/256/0/180] [15037.129313] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15037.170484] LNet: Added LNI 192.168.201.118@tcp22 [8/256/0/180] [15044.665211] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15044.703718] LNet: Added LNI 192.168.201.118@tcp23 [8/256/0/180] [15052.276861] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15052.303094] LNet: Added LNI 192.168.201.118@tcp24 [8/256/0/180] [15058.490254] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15058.515237] LNet: Added LNI 192.168.201.118@tcp25 [8/256/0/180] [15066.812622] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15066.869942] LNet: Added LNI 192.168.201.118@tcp26 [8/256/0/180] [15074.061411] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15074.109373] LNet: Added LNI 192.168.201.118@tcp27 [8/256/0/180] [15080.892235] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15080.925985] LNet: Added LNI 192.168.201.118@tcp28 [8/256/0/180] [15087.340501] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15087.399232] LNet: Added LNI 192.168.201.118@tcp29 [8/256/0/180] [15096.876711] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15096.931550] LNet: Added LNI 192.168.201.118@tcp30 [8/256/0/180] [15106.598532] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15106.667533] LNet: Added LNI 192.168.201.118@tcp31 [8/256/0/180] [15115.264834] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15115.303125] LNet: Added LNI 192.168.201.118@tcp32 [8/256/0/180] [15124.492123] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15124.524406] LNet: Added LNI 192.168.201.118@tcp33 [8/256/0/180] [15133.472933] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15133.514304] LNet: Added LNI 192.168.201.118@tcp34 [8/256/0/180] [15142.666378] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15142.708625] LNet: Added LNI 192.168.201.118@tcp35 [8/256/0/180] [15151.035826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15151.078627] LNet: Added LNI 192.168.201.118@tcp36 [8/256/0/180] [15160.605570] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15160.662178] LNet: Added LNI 192.168.201.118@tcp37 [8/256/0/180] [15171.734800] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15171.791562] LNet: Added LNI 192.168.201.118@tcp38 [8/256/0/180] [15182.262930] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15182.310995] LNet: Added LNI 192.168.201.118@tcp39 [8/256/0/180] [15192.680420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15192.713806] LNet: Added LNI 192.168.201.118@tcp40 [8/256/0/180] [15202.299692] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15202.344368] LNet: Added LNI 192.168.201.118@tcp41 [8/256/0/180] [15212.751693] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15212.821218] LNet: Added LNI 192.168.201.118@tcp42 [8/256/0/180] [15222.654632] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15222.700304] LNet: Added LNI 192.168.201.118@tcp43 [8/256/0/180] [15233.438633] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15233.474440] LNet: Added LNI 192.168.201.118@tcp44 [8/256/0/180] [15243.736164] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15243.791265] LNet: Added LNI 192.168.201.118@tcp45 [8/256/0/180] [15254.217831] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15254.258564] LNet: Added LNI 192.168.201.118@tcp46 [8/256/0/180] [15264.712761] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15264.750524] LNet: Added LNI 192.168.201.118@tcp47 [8/256/0/180] [15274.879107] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15274.915989] LNet: Added LNI 192.168.201.118@tcp48 [8/256/0/180] [15283.300585] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15283.328752] LNet: Added LNI 192.168.201.118@tcp49 [8/256/0/180] [15294.764242] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15294.826244] LNet: Added LNI 192.168.201.118@tcp50 [8/256/0/180] [15318.509464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15318.517297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15318.545246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15321.577270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15321.579549] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15321.603952] Lustre: Skipped 1 previous similar message [15321.626755] Lustre: Skipped 1 previous similar message [15326.689734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15326.696852] Lustre: Skipped 2 previous similar messages [15331.295134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15331.978411] Lustre: server umount lustre-MDT0000 complete [15336.929907] LustreError: 473282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15336.957237] LustreError: 473282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [15336.987283] LustreError: 474067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492649 with bad export cookie 1656661085101412311 [15336.992398] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15337.001293] LustreError: 474067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15337.548416] Lustre: server umount lustre-MDT0001 complete [15358.431427] Lustre: 472543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789492654/real 1789492654] req@00000000549eb8ad x1876418541850240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789492670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15358.475902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15359.835552] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [15363.551290] Lustre: 472544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789492659/real 1789492659] req@0000000080d81e21 x1876418541850368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789492675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15368.678489] Lustre: 472545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789492664/real 1789492664] req@00000000aab7863d x1876418541850624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789492680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15370.236694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15372.833431] Lustre: 472545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789492669/real 1789492669] req@0000000096c0ac35 x1876418541850752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789492685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15382.115775] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x16fda47ac2aeda50 [15382.131614] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [15382.554788] LustreError: 475236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15382.649387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15383.648277] LustreError: 499498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15383.670880] LustreError: 499498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15388.661455] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15389.155647] LustreError: 499495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15389.196374] LustreError: 499495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15392.249562] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15397.348085] LustreError: 475237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15397.382049] LustreError: 475237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [15398.342597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15398.692943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15404.006357] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15404.038750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15412.736831] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15420.012857] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15443.949497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15443.961389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15443.995853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15444.965104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15449.954326] Lustre: server umount lustre-OST0000 complete [15455.207877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15455.215703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15455.216064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15455.239754] Lustre: Skipped 2 previous similar messages [15455.289877] Lustre: Skipped 3 previous similar messages [15461.197690] Lustre: server umount lustre-MDT0000 complete [15465.442419] LustreError: 499495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15465.477901] LustreError: 499495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15465.832766] LustreError: 474067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492778 with bad export cookie 1656661085101414992 [15465.835463] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15465.842400] LustreError: 474067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15466.260986] Lustre: server umount lustre-MDT0001 complete [15479.583631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [15482.979908] Key type lgssc unregistered [15483.509534] LNet: 502300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15483.518258] LNetError: 502300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15483.527363] LNet: Removed LNI 192.168.201.118@tcp [15483.529225] LNet: Removed LNI 192.168.201.118@tcp10 [15483.537117] LNet: Removed LNI 192.168.201.118@tcp11 [15483.543618] LNet: Removed LNI 192.168.201.118@tcp12 [15483.556294] LNet: Removed LNI 192.168.201.118@tcp13 [15483.561447] LNet: Removed LNI 192.168.201.118@tcp14 [15483.573517] LNet: Removed LNI 192.168.201.118@tcp15 [15483.577233] LNet: Removed LNI 192.168.201.118@tcp16 [15483.584827] LNet: Removed LNI 192.168.201.118@tcp17 [15483.586814] LNet: Removed LNI 192.168.201.118@tcp18 [15483.594786] LNet: Removed LNI 192.168.201.118@tcp19 [15483.602779] LNet: Removed LNI 192.168.201.118@tcp20 [15483.608594] LNet: Removed LNI 192.168.201.118@tcp21 [15483.613264] LNet: Removed LNI 192.168.201.118@tcp22 [15483.627128] LNet: Removed LNI 192.168.201.118@tcp23 [15483.633680] LNet: Removed LNI 192.168.201.118@tcp24 [15483.637261] LNet: Removed LNI 192.168.201.118@tcp25 [15483.641630] LNet: Removed LNI 192.168.201.118@tcp26 [15483.646552] LNet: Removed LNI 192.168.201.118@tcp27 [15483.649186] LNet: Removed LNI 192.168.201.118@tcp28 [15483.654695] LNet: Removed LNI 192.168.201.118@tcp29 [15483.661219] LNet: Removed LNI 192.168.201.118@tcp30 [15483.666855] LNet: Removed LNI 192.168.201.118@tcp31 [15483.672725] LNet: Removed LNI 192.168.201.118@tcp32 [15483.677945] LNet: Removed LNI 192.168.201.118@tcp33 [15483.682865] LNet: Removed LNI 192.168.201.118@tcp34 [15483.687812] LNet: Removed LNI 192.168.201.118@tcp35 [15483.692720] LNet: Removed LNI 192.168.201.118@tcp36 [15483.697850] LNet: Removed LNI 192.168.201.118@tcp37 [15483.705833] LNet: Removed LNI 192.168.201.118@tcp38 [15483.711716] LNet: Removed LNI 192.168.201.118@tcp39 [15483.716850] LNet: Removed LNI 192.168.201.118@tcp40 [15483.721731] LNet: Removed LNI 192.168.201.118@tcp41 [15483.730724] LNet: Removed LNI 192.168.201.118@tcp42 [15483.734956] LNet: Removed LNI 192.168.201.118@tcp43 [15483.741209] LNet: Removed LNI 192.168.201.118@tcp44 [15483.746295] LNet: Removed LNI 192.168.201.118@tcp45 [15483.752534] LNet: Removed LNI 192.168.201.118@tcp46 [15483.764581] LNet: Removed LNI 192.168.201.118@tcp47 [15483.769477] LNet: Removed LNI 192.168.201.118@tcp48 [15483.776124] LNet: Removed LNI 192.168.201.118@tcp49 [15483.800852] LNet: Removed LNI 192.168.201.118@tcp50 [15484.897899] Key type .llcrypt unregistered [15484.899878] Key type ._llcrypt unregistered [15508.677087] Key type ._llcrypt registered [15508.686332] Key type .llcrypt registered [15508.884314] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [15530.607231] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [15532.487290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15532.545352] alg: No test for adler32 (adler32-zlib) [15533.721653] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [15534.099665] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [15535.839841] Key type lgssc registered [15537.587496] Lustre: Echo OBD driver; http://www.lustre.org/ [15577.999511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15578.039396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15579.374259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15579.425451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15579.591049] Lustre: lustre-MDT0000: new disk, initializing [15579.737886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15579.761590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15584.545458] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15597.504810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15597.738310] Lustre: 505499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15597.925790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15597.932839] Lustre: Skipped 1 previous similar message [15598.168844] Lustre: lustre-MDT0001: new disk, initializing [15598.390214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15598.469516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15598.488460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15604.572606] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15616.891227] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15623.020496] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15632.268862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15632.506470] Lustre: *** cfs_fail_loc=90f, val=0*** [15632.618529] Lustre: lustre-OST0000: new disk, initializing [15632.622446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15632.637943] Lustre: 507517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15632.745881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15639.137706] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15642.707401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15642.725455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15642.890705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15651.162734] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15663.585428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15663.588894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15663.594084] LustreError: Skipped 1 previous similar message [15663.604317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15663.630365] Lustre: Skipped 1 previous similar message [15665.921781] Lustre: server umount lustre-OST0000 complete [15673.828444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15673.842537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15673.845376] Lustre: Skipped 1 previous similar message [15677.247769] Lustre: server umount lustre-MDT0000 complete [15681.338413] LustreError: 505491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492993 with bad export cookie 17672677734227110761 [15681.349833] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15681.360604] LustreError: 505491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15682.249433] Lustre: server umount lustre-MDT0001 complete [15695.255845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [15698.931810] Key type lgssc unregistered [15699.275390] LNet: 509427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15699.282917] LNetError: 509427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15700.330094] LNet: Removed LNI 192.168.201.118@tcp [15701.674814] Key type .llcrypt unregistered [15701.676532] Key type ._llcrypt unregistered [15725.991420] Key type ._llcrypt registered [15725.994325] Key type .llcrypt registered [15726.117447] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [15728.130572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15728.172271] alg: No test for adler32 (adler32-zlib) [15729.449931] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [15729.913785] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [15731.759391] Key type lgssc registered [15733.416517] Lustre: Echo OBD driver; http://www.lustre.org/ [15744.314365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15744.343737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15746.016151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15752.346788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15756.263630] LustreError: 510701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15756.292901] LustreError: 510701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15761.389280] LustreError: 510702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15762.812626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15763.378790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15768.615704] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15777.245347] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15784.142645] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15791.354815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15791.654070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15798.919713] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15807.606869] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15830.544426] loop0: detected capacity change from 0 to 204800000 [15830.721654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15830.733689] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15830.742076] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15838.133585] loop0: detected capacity change from 0 to 204800000 [15838.524724] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15838.535589] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15838.545437] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15844.671423] loop0: detected capacity change from 0 to 204800000 [15851.224532] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15851.511148] Lustre: fs15246-OST0000: new disk, initializing [15851.523568] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15851.532131] Lustre: 514317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15851.592239] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15859.452951] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15870.622376] loop1: detected capacity change from 0 to 204800000 [15877.307443] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15877.499814] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15877.557195] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15877.661980] Lustre: fs15246-MDT0000: new disk, initializing [15877.807635] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15877.838664] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15877.960632] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15879.008907] Lustre: 515439:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000ad7575c x1876419456061184/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789493207 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15883.543728] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [15884.141049] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15884.156286] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15884.195111] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15909.349876] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15909.365693] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15914.470976] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15915.242247] Lustre: server umount fs15246-MDT0000 complete [15919.609488] LustreError: 510696:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15924.713911] LustreError: 512791:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15927.326523] Lustre: server umount fs15246-OST0000 complete [15940.067173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15940.089993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15940.098119] Lustre: Skipped 1 previous similar message [15945.202579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15945.219552] Lustre: Skipped 1 previous similar message [15946.090617] Lustre: server umount lustre-OST0000 complete [15950.324733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15950.325736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15950.341570] Lustre: Skipped 2 previous similar messages [15950.367263] Lustre: Skipped 1 previous similar message [15956.459207] Lustre: server umount lustre-MDT0000 complete [15960.545684] LustreError: 510701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15960.571335] LustreError: 510701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15961.379795] LustreError: 510681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789493273 with bad export cookie 17607936090970232251 [15961.381171] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15961.398476] LustreError: 510681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15961.816511] Lustre: server umount lustre-MDT0001 complete [15976.049420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [15979.514508] Key type lgssc unregistered [15979.816748] LNet: 518541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15979.834549] LNetError: 518541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15979.862699] LNet: Removed LNI 192.168.201.118@tcp [15981.059865] Key type .llcrypt unregistered [15981.066411] Key type ._llcrypt unregistered [16004.183951] Key type ._llcrypt registered [16004.204656] Key type .llcrypt registered [16004.446535] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [16006.495972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16006.526365] alg: No test for adler32 (adler32-zlib) [16007.777362] Lustre: Lustre: Build Version: 2.17.57_84_gd41d3c6 [16008.111971] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [16009.871460] Key type lgssc registered [16011.628218] Lustre: Echo OBD driver; http://www.lustre.org/ [16064.750459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [16075.947520] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [16120.253287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16120.271337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16121.666728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16121.716284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16121.823761] Lustre: lustre-MDT0000: new disk, initializing [16121.942828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16121.956146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16127.322901] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [16138.772517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16138.969889] Lustre: 523011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16139.042989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16139.046583] Lustre: Skipped 1 previous similar message [16139.124702] Lustre: lustre-MDT0001: new disk, initializing [16139.227838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16139.302480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16139.321831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16145.170899] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [16155.303348] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16161.263555] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16168.747625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16169.134970] Lustre: lustre-OST0000: new disk, initializing [16169.140819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16169.165745] Lustre: 524979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16169.320236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16175.499646] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [16177.242918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16177.253041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16177.419660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16189.412556] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16195.925853] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16196.138580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16200.975934] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16201.169304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16203.233496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16203.235243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16203.249989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16203.262755] Lustre: Skipped 1 previous similar message [16208.362121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16208.372067] Lustre: Skipped 1 previous similar message [16209.455557] Lustre: server umount lustre-OST0000 complete [16213.473818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16213.478838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16213.492062] Lustre: Skipped 2 previous similar messages [16218.594330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16218.603780] Lustre: Skipped 2 previous similar messages [16220.011619] Lustre: server umount lustre-MDT0000 complete [16224.195188] LustreError: 523003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789493536 with bad export cookie 12730820972834147042 [16224.202056] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16224.207910] LustreError: 523003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16224.687420] Lustre: server umount lustre-MDT0001 complete [16249.287051] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [16260.540648] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [16296.101154] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16051 sec ========= 13:33:27 (1789493607) [16298.233665] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:33:29 (1789493609) === [16302.666740] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:33:33 (1789493613) === [16349.961619] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [16353.553908] Key type lgssc unregistered [16353.883752] LNet: 531629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16353.891619] LNetError: 531629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16353.907550] LNet: Removed LNI 192.168.201.118@tcp [16355.066360] Key type .llcrypt unregistered [16355.071561] Key type ._llcrypt unregistered