[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 675234669 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002400] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010068] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.020311] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025321] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] *** VALIDATE tmpfs *** [ 0.030770] *** VALIDATE proc *** [ 0.031296] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041460] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.043289] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044753] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052331] rcu: Hierarchical SRCU implementation. [ 0.054564] smp: Bringing up secondary CPUs ... [ 0.056082] x86: Booting SMP configuration: [ 0.057014] .... node #0, CPUs: #1 #2 #3 [ 0.071335] smp: Brought up 1 node, 4 CPUs [ 0.073014] smpboot: Max logical packages: 1 [ 0.074020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.115064] node 0 deferred pages initialised in 39ms [ 0.120213] devtmpfs: initialized [ 0.121316] x86/mm: Memory block size: 128MB [ 0.123904] gcov: version magic: 0x41383552 [ 0.125144] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127243] pinctrl core: initialized pinctrl subsystem [ 0.128277] [ 0.129008] ************************************************************* [ 0.130014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131014] ** ** [ 0.132014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133012] ** ** [ 0.134014] ** This means that this kernel is built to expose internal ** [ 0.135014] ** IOMMU data structures, which may compromise security on ** [ 0.136014] ** your system. ** [ 0.137013] ** ** [ 0.138015] ** If you see this message and you are not debugging the ** [ 0.139014] ** kernel, report this immediately to your vendor! ** [ 0.140013] ** ** [ 0.141012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142014] ************************************************************* [ 0.143771] NET: Registered protocol family 16 [ 0.144425] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.145056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.146063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.147428] cpuidle: using governor menu [ 0.150033] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.155775] PCI: Using configuration type 1 for base access [ 0.159184] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170167] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.172073] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174640] cryptd: max_cpu_qlen set to 1000 [ 0.177426] ACPI: Added _OSI(Module Device) [ 0.179017] ACPI: Added _OSI(Processor Device) [ 0.181056] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189531] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197008] ACPI: Interpreter enabled [ 0.199124] ACPI: PM: (supports S0 S3 S4 S5) [ 0.201014] ACPI: Using IOAPIC for interrupt routing [ 0.203156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209404] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222638] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242308] acpiphp: Slot [2] registered [ 0.244213] acpiphp: Slot [5] registered [ 0.246124] acpiphp: Slot [6] registered [ 0.248125] acpiphp: Slot [7] registered [ 0.250136] acpiphp: Slot [8] registered [ 0.252150] acpiphp: Slot [9] registered [ 0.253302] acpiphp: Slot [10] registered [ 0.256324] acpiphp: Slot [3] registered [ 0.258109] acpiphp: Slot [4] registered [ 0.261127] acpiphp: Slot [11] registered [ 0.263114] acpiphp: Slot [12] registered [ 0.265141] acpiphp: Slot [13] registered [ 0.267149] acpiphp: Slot [14] registered [ 0.269125] acpiphp: Slot [15] registered [ 0.271129] acpiphp: Slot [16] registered [ 0.273176] acpiphp: Slot [17] registered [ 0.276114] acpiphp: Slot [18] registered [ 0.278141] acpiphp: Slot [19] registered [ 0.280439] acpiphp: Slot [20] registered [ 0.283118] acpiphp: Slot [21] registered [ 0.285115] acpiphp: Slot [22] registered [ 0.287116] acpiphp: Slot [23] registered [ 0.288000] acpiphp: Slot [24] registered [ 0.288000] acpiphp: Slot [25] registered [ 0.290117] acpiphp: Slot [26] registered [ 0.292125] acpiphp: Slot [27] registered [ 0.294093] acpiphp: Slot [28] registered [ 0.296112] acpiphp: Slot [29] registered [ 0.298116] acpiphp: Slot [30] registered [ 0.300110] acpiphp: Slot [31] registered [ 0.302102] PCI host bridge to bus 0000:00 [ 0.304021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.313026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327290] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331774] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.359019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.366681] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370959] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402511] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.478937] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.490014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.500015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.595025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.624463] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.647024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.664015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.701015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.715470] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.723016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.731016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.748056] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.757336] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.764017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.774016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.799017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.812270] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.820015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.830025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.852022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.867099] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.869368] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.875408] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.877361] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.880239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.885066] iommu: Default domain type: Passthrough [ 0.886528] SCSI subsystem initialized [ 0.889146] ACPI: bus type USB registered [ 0.890150] usbcore: registered new interface driver usbfs [ 0.893106] usbcore: registered new interface driver hub [ 0.895073] usbcore: registered new device driver usb [ 0.897178] pps_core: LinuxPPS API ver. 1 registered [ 0.899013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.902060] PTP clock support registered [ 0.904410] EDAC MC: Ver: 3.0.0 [ 0.905143] PCI: Using ACPI for IRQ routing [ 0.906000] NetLabel: Initializing [ 0.906000] NetLabel: domain hash size = 128 [ 0.908014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.910098] NetLabel: unlabeled traffic allowed by default [ 0.912219] vgaarb: loaded [ 0.913335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.923265] clocksource: Switched to clocksource kvm-clock [ 1.163839] VFS: Disk quotas dquot_6.6.0 [ 1.165977] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.169478] *** VALIDATE ramfs *** [ 1.171362] *** VALIDATE hugetlbfs *** [ 1.173409] pnp: PnP ACPI init [ 1.176713] pnp: PnP ACPI: found 6 devices [ 1.196671] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.202654] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.209112] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.214467] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.220520] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.227486] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.234675] NET: Registered protocol family 2 [ 1.250280] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.292924] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.297107] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.303179] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.307071] TCP: Hash tables configured (established 65536 bind 65536) [ 1.310507] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.316528] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.320449] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.323583] NET: Registered protocol family 1 [ 1.326412] RPC: Registered named UNIX socket transport module. [ 1.328586] RPC: Registered udp transport module. [ 1.330723] RPC: Registered tcp transport module. [ 1.332581] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.335462] NET: Registered protocol family 44 [ 1.337749] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.340251] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.342233] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.349544] PCI: CLS 0 bytes, default 64 [ 1.353971] Unpacking initramfs... [ 3.253937] debug: unmapping init [mem 0xffff94dfbcc54000-0xffff94dfbffbffff] [ 3.261839] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.265040] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.270680] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.582292] Initialise system trusted keyrings [ 4.585312] Key type blacklist registered [ 4.594686] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.614131] zbud: loaded [ 4.621559] *** VALIDATE nfs *** [ 4.623551] *** VALIDATE nfs4 *** [ 4.627754] pstore: using deflate compression [ 4.633152] Platform Keyring initialized [ 4.843636] NET: Registered protocol family 38 [ 4.845959] Key type asymmetric registered [ 4.848543] Asymmetric key parser 'x509' registered [ 4.851340] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.855874] io scheduler mq-deadline registered [ 4.858242] io scheduler kyber registered [ 4.860781] io scheduler bfq registered [ 4.863658] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.866867] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.870839] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.875362] ACPI: Power Button [PWRF] [ 4.883759] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.892980] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.993224] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.002256] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.020638] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.055925] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.087039] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.092598] Non-volatile memory driver v1.3 [ 5.094744] Linux agpgart interface v0.103 [ 5.135479] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.140720] vda: detected capacity change from 0 to 74698752 [ 5.171682] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.177564] vdb: detected capacity change from 0 to 1073741824 [ 5.216758] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.220457] vdc: detected capacity change from 0 to 2621440000 [ 5.245468] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.248962] vdd: detected capacity change from 0 to 2621440000 [ 5.269709] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.275335] vde: detected capacity change from 0 to 4294967296 [ 5.298212] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.302918] vdf: detected capacity change from 0 to 4294967296 [ 5.312740] libphy: Fixed MDIO Bus: probed [ 5.322621] usbcore: registered new interface driver usbserial_generic [ 5.326672] usbserial: USB Serial support registered for generic [ 5.329632] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.335968] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.338649] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.342598] mousedev: PS/2 mouse device common for all mice [ 5.347450] rtc_cmos 00:05: RTC can wake from S4 [ 5.355944] rtc_cmos 00:05: registered as rtc0 [ 5.357505] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.358874] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.358926] intel_pstate: CPU model not supported [ 5.361821] hid: raw HID events driver (C) Jiri Kosina [ 5.371975] usbcore: registered new interface driver usbhid [ 5.374933] usbhid: USB HID core driver [ 5.377520] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.381895] drop_monitor: Initializing network drop monitor service [ 5.395329] Initializing XFRM netlink socket [ 5.399082] NET: Registered protocol family 10 [ 5.405795] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.407868] Segment Routing with IPv6 [ 5.440957] NET: Registered protocol family 17 [ 5.543364] mpls_gso: MPLS GSO support [ 5.574252] RAS: Correctable Errors collector initialized. [ 5.583306] AVX version of gcm_enc/dec engaged. [ 5.589655] AES CTR mode by8 optimization enabled [ 5.737567] sched_clock: Marking stable (5737545845, 0)->(7093367201, -1355821356) [ 5.741774] registered taskstats version 1 [ 5.744314] Loading compiled-in X.509 certificates [ 5.747170] zswap: loaded using pool lzo/zbud [ 5.788212] Key type big_key registered [ 5.805716] Key type encrypted registered [ 5.807702] ima: No TPM chip found, activating TPM-bypass! [ 5.810516] ima: Allocated hash algorithm: sha1 [ 5.812572] ima: No architecture policies found [ 5.814826] evm: Initialising EVM extended attributes: [ 5.816989] evm: security.selinux [ 5.818526] evm: security.ima [ 5.819803] evm: security.capability [ 5.821186] evm: HMAC attrs: 0x1 [ 5.824191] rtc_cmos 00:05: setting system clock to 2026-08-08 09:59:16 UTC (1786183156) [ 5.832978] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 5.836993] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 5.846240] Write protecting the kernel read-only data: 28672k [ 5.850462] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 5.853549] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 5.909643] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.918239] systemd[1]: Detected virtualization kvm. [ 5.919786] systemd[1]: Detected architecture x86-64. [ 5.921514] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.954329] systemd[1]: No hostname configured. [ 5.957274] systemd[1]: Set hostname to . [ 5.963467] random: systemd: uninitialized urandom read (16 bytes read) [ 5.965929] systemd[1]: Initializing machine ID from random generator. [ 6.083687] random: ln: uninitialized urandom read (6 bytes read) [ 6.246958] random: systemd: uninitialized urandom read (16 bytes read) [ 6.252917] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.272892] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 6.299857] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.588544] device-mapper: uevent: version 1.0.3 [ 8.591621] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 10.339790] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.786496] random: fast init done [ 11.301700] scsi host0: ata_piix [ 11.464542] scsi host1: ata_piix [ 11.466301] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.474990] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.373369] random: crng init done [ 16.374952] random: 7 urandom warning(s) missed due to ratelimiting [ 19.290989] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.734950] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.565526] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.306622] SELinux: Disabled at runtime. [ 25.423972] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.438889] systemd[1]: Detected virtualization kvm. [ 25.442263] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.596191] hrtimer: interrupt took 2512096 ns [ 26.910736] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.917795] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.929786] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.939845] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.948623] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.965751] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.988668] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 27.765769] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 28.979752] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.067304] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.144958] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.221767] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.513701] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 37.805475] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 38.859592] NFS: Registering the id_resolver key type [ 38.861819] Key type id_resolver registered [ 38.863776] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg310-server login: [ 101.659798] libcfs: loading out-of-tree module taints kernel. [ 101.738379] Key type ._llcrypt registered [ 101.741144] Key type .llcrypt registered [ 101.853154] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 121.294390] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 124.094449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.109236] alg: No test for adler32 (adler32-zlib) [ 125.897997] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 126.832883] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 128.520343] Key type lgssc registered [ 130.817336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.650143] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.188930] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 205.834686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 205.851580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.053466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 207.085744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 207.145438] Lustre: lustre-MDT0000: new disk, initializing [ 207.216599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.240745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 212.390129] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.375485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.484406] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.564379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.570424] Lustre: Skipped 1 previous similar message [ 226.706613] Lustre: lustre-MDT0001: new disk, initializing [ 226.838753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.887880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 226.931937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.204301] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.168876] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.344788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.864681] Lustre: lustre-OST0000: new disk, initializing [ 249.869636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.881642] Lustre: 8470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.953060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.689292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 256.700697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.791033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.180994] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.709146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.832675] Lustre: lustre-OST0001: new disk, initializing [ 273.835779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.841954] Lustre: 9546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.898029] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.933104] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.716893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 282.722588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 282.813081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 291.988609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.477229] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.682216] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing check_logdir /tmp/testlogs/ [ 311.500507] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing yml_node [ 316.835780] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 319.714430] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 322.231584] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 324.066845] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 06:04:33 EDT 2026 [ 341.840402] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 343.237984] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.400827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.405221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.423580] Lustre: Skipped 3 previous similar messages [ 359.445372] Lustre: Skipped 3 previous similar messages [ 366.489877] Lustre: server umount lustre-MDT0000 complete [ 369.639175] LustreError: 9004:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.670605] LustreError: 9004:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 8 previous similar messages [ 371.192293] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183521 with bad export cookie 6266496135228750092 [ 371.207482] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.237214] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 374.753360] LustreError: 6539:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.753856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.762258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.771699] LustreError: 6539:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 374.811521] Lustre: Skipped 1 previous similar message [ 376.802519] LustreError: 9004:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.802538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.802549] Lustre: Skipped 1 previous similar message [ 376.815377] LustreError: 9004:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 377.578411] Lustre: server umount lustre-MDT0001 complete [ 381.515947] Lustre: server umount lustre-OST0000 complete [ 385.488474] Lustre: server umount lustre-OST0001 complete [ 393.400622] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 404.277411] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 443.315350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.620320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.708866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.769460] Lustre: lustre-MDT0000: new disk, initializing [ 443.891467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.927495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.757318] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 458.736495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.935675] Lustre: 15109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.065873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.070063] Lustre: Skipped 1 previous similar message [ 459.139217] Lustre: lustre-MDT0001: new disk, initializing [ 459.227375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.268256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 459.278692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 463.892806] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 474.229941] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.216845] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 487.076449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.424088] Lustre: lustre-OST0000: new disk, initializing [ 487.427430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 487.433514] Lustre: 17079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 487.536694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.698439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 488.707455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 489.192714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 494.977331] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 505.241766] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.818865] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.042477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 515.638994] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 515.796742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 519.649500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.656121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.671039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.680993] Lustre: Skipped 1 previous similar message [ 524.084806] Lustre: server umount lustre-OST0000 complete [ 529.898230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.906427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.949908] Lustre: Skipped 2 previous similar messages [ 529.965388] Lustre: Skipped 2 previous similar messages [ 534.345362] Lustre: server umount lustre-MDT0000 complete [ 535.015543] LustreError: 15120:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.083143] LustreError: 15120:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 538.976398] LustreError: 15102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183689 with bad export cookie 6266496135228751632 [ 538.979624] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.989448] LustreError: 15102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 539.206331] Lustre: server umount lustre-MDT0001 complete [ 543.845688] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:08:13 (1786183693) [ 546.993230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 549.139108] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:08:18 (1786183698) [ 556.267789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.758211] LustreError: 19489:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.880306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.866679] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 567.278615] LustreError: 19490:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.308554] LustreError: 19490:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 571.579766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.168258] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 583.594871] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 589.969547] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.574390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 606.649989] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 614.813276] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 627.171829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 627.188939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.197848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.217296] Lustre: Skipped 1 previous similar message [ 627.250765] Lustre: Skipped 1 previous similar message [ 631.728458] Lustre: server umount lustre-OST0000 complete [ 637.409727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.421595] Lustre: Skipped 1 previous similar message [ 641.322678] Lustre: server umount lustre-MDT0000 complete [ 642.533162] LustreError: 19484:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.561515] LustreError: 19484:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 646.083206] LustreError: 22271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183796 with bad export cookie 6266496135228752514 [ 646.085712] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.091018] LustreError: 22271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 646.513501] Lustre: server umount lustre-MDT0001 complete [ 661.942237] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 664.460698] Key type lgssc unregistered [ 664.717630] LNet: 23026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.723809] LNetError: 23026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.741962] LNet: Removed LNI 192.168.203.110@tcp [ 665.641155] Key type .llcrypt unregistered [ 665.643324] Key type ._llcrypt unregistered [ 690.479770] Key type ._llcrypt registered [ 690.483585] Key type .llcrypt registered [ 690.555386] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 692.568626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.742658] alg: No test for adler32 (adler32-zlib) [ 695.199810] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 695.524706] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 697.160171] Key type lgssc registered [ 698.083459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 705.764795] loop: module loaded [ 705.812132] loop0: detected capacity change from 0 to 204800000 [ 706.134267] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.151514] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.507585] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.866436] loop0: detected capacity change from 0 to 204800000 [ 713.254220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.268924] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.282287] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 718.733544] loop0: detected capacity change from 0 to 204800000 [ 724.942991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 724.983892] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.247201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 726.300687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 726.352477] Lustre: lustre-MDT0000: new disk, initializing [ 726.411127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.451071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 731.531578] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 741.418435] loop1: detected capacity change from 0 to 204800000 [ 747.785836] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.114322] Lustre: lustre-OST07c6: new disk, initializing [ 748.120729] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 748.130335] Lustre: Skipped 1 previous similar message [ 748.136580] Lustre: 26213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 748.208522] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 753.897624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 753.949971] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 753.992498] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 756.076737] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 762.639634] Lustre: 25200:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 762.652193] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.657150] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.665281] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.671674] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 762.677410] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 782.057980] Lustre: 26818:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 782.066847] Lustre: 26818:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 782.076710] Lustre: 26818:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 782.089044] Lustre: 26818:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 782.099646] Lustre: 26818:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 782.111766] Lustre: 26818:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.908490] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 786.916719] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 786.921853] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 786.929584] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.936056] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.942594] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.948336] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 786.955708] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.962361] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.967962] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.972978] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.978816] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.398183] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 790.410251] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 790.417177] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 790.420572] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.425872] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.429949] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.433544] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 790.437890] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.441929] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 790.445911] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.450642] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.454214] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.270191] Lustre: 25200:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 795.275229] Lustre: 25200:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 795.279281] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 795.283577] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.288613] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 795.302469] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.309296] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 795.316349] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.322262] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 795.328285] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.334298] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 795.339715] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.508221] Lustre: 26818:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 806.516341] Lustre: 26818:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 806.520893] Lustre: 26818:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 806.525500] Lustre: 26818:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 806.530557] Lustre: 26818:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.535661] Lustre: 26818:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 806.540875] Lustre: 26818:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 806.546171] Lustre: 26818:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 806.550903] Lustre: 26818:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 806.555311] Lustre: 26818:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 806.560329] Lustre: 26818:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.565846] Lustre: 26818:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 856.045208] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.069593] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 859.095852] Lustre: server umount lustre-OST07c6 complete [ 868.659702] Lustre: server umount lustre-MDT0000 complete [ 886.692866] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 888.756954] Key type lgssc unregistered [ 888.956159] LNet: 28772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.962847] LNetError: 28772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.979440] LNet: Removed LNI 192.168.203.110@tcp [ 890.010234] Key type .llcrypt unregistered [ 890.013993] Key type ._llcrypt unregistered [ 909.830422] Key type ._llcrypt registered [ 909.832675] Key type .llcrypt registered [ 909.919980] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 910.792513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.987279] alg: No test for adler32 (adler32-zlib) [ 912.916538] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 913.180336] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 914.952244] Key type lgssc registered [ 916.121355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 923.092804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 923.109684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.410418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.966294] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 934.904985] LustreError: 30042:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.922465] LustreError: 30042:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 938.132635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.377384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 943.155884] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 952.615270] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 958.200287] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 963.658167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.954212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.105105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 970.796214] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 977.669535] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 997.874463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.886966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.887138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 997.903589] Lustre: Skipped 1 previous similar message [ 999.090472] Lustre: server umount lustre-OST0000 complete [ 1005.028552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1005.035739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.051318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.059620] Lustre: Skipped 1 previous similar message [ 1008.106790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.112534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.441692] Lustre: server umount lustre-MDT0000 complete [ 1013.288456] LustreError: 30042:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.783801] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184165 with bad export cookie 6344717475421525793 [ 1014.797169] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1014.807654] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1015.110375] Lustre: server umount lustre-MDT0001 complete [ 1028.090755] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 1031.315116] Key type lgssc unregistered [ 1031.542768] LNet: 33782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.554676] LNetError: 33782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.579643] LNet: Removed LNI 192.168.203.110@tcp [ 1032.470281] Key type .llcrypt unregistered [ 1032.473127] Key type ._llcrypt unregistered [ 1052.218428] Key type ._llcrypt registered [ 1052.223365] Key type .llcrypt registered [ 1052.557031] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1054.801182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.055840] alg: No test for adler32 (adler32-zlib) [ 1056.352592] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1056.730638] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 1058.608161] Key type lgssc registered [ 1059.649806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.371388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.385757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.828068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.621441] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1079.266409] LustreError: 35053:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.281819] LustreError: 35053:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1082.086401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.398421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.402189] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1093.696349] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1098.231357] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1106.311116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1106.627539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.898466] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1113.841518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1121.230878] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.538585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1133.542173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.551623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.582395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1134.590127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.593380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.611836] Lustre: Skipped 1 previous similar message [ 1136.747459] Lustre: server umount lustre-OST0000 complete [ 1143.777868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1143.782095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.789960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.958481] Lustre: server umount lustre-MDT0000 complete [ 1149.926019] LustreError: 35048:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.948466] LustreError: 35048:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1150.998851] LustreError: 35032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184301 with bad export cookie 13353285279345174574 [ 1151.008868] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.010842] LustreError: 35032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.466319] Lustre: server umount lustre-MDT0001 complete [ 1162.851439] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 1165.693934] Key type lgssc unregistered [ 1166.050689] LNet: 38588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.069955] LNetError: 38588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.086762] LNet: Removed LNI 192.168.203.110@tcp [ 1167.052143] Key type .llcrypt unregistered [ 1167.053915] Key type ._llcrypt unregistered [ 1192.069846] Key type ._llcrypt registered [ 1192.072712] Key type .llcrypt registered [ 1192.203987] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1194.974091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1195.015120] alg: No test for adler32 (adler32-zlib) [ 1195.982544] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1196.189852] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 1197.856621] Key type lgssc registered [ 1199.164582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.076902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.092680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.474083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.813544] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1219.042947] LustreError: 40006:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.088674] LustreError: 40006:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1220.067639] LustreError: 40005:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.399309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.722268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.042682] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1235.206351] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1242.841226] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.326049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1250.586014] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.708582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1257.432465] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1264.644922] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.626040] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 1279.461600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.466252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1279.478646] Lustre: Skipped 2 previous similar messages [ 1279.486044] Lustre: Skipped 2 previous similar messages [ 1281.306304] Lustre: server umount lustre-MDT0000 complete [ 1282.853100] LustreError: 41438:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.793711] LustreError: 41848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184435 with bad export cookie 10092068547331756094 [ 1284.809640] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.813885] LustreError: 41848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1288.375881] LustreError: 40001:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.387278] LustreError: 40001:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 1288.402731] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.10@tcp (stopping) [ 1289.699578] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.189721] Lustre: server umount lustre-MDT0001 complete [ 1312.568474] Lustre: server umount lustre-OST0000 complete [ 1329.443477] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 1332.695787] Key type lgssc unregistered [ 1332.931628] LNet: 43870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.937236] LNetError: 43870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.953892] LNet: Removed LNI 192.168.203.110@tcp [ 1333.745828] Key type .llcrypt unregistered [ 1333.747820] Key type ._llcrypt unregistered [ 1357.867318] Key type ._llcrypt registered [ 1357.869752] Key type .llcrypt registered [ 1357.947847] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1360.674266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1360.809260] alg: No test for adler32 (adler32-zlib) [ 1362.074509] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1362.346484] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 1364.113962] Key type lgssc registered [ 1364.994750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1372.997651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.054493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.476400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.575883] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1384.931775] LustreError: 45288:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.941190] LustreError: 45288:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1387.996669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.317618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1392.762856] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1401.229110] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1406.417200] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1412.800855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.082852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.666649] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1420.337168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1425.560044] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1433.280729] Lustre: lustre-OST0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 1434.604140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.608805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.615111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.619330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.621096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.623502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1438.382993] Lustre: lustre-OST0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 1438.402704] Lustre: Skipped 1 previous similar message [ 1443.538674] Lustre: lustre-OST0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 1443.545334] Lustre: Skipped 2 previous similar messages [ 1446.368464] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1446.468378] Lustre: server umount lustre-OST0000 complete [ 1463.661387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.685625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.690989] Lustre: Skipped 3 previous similar messages [ 1469.977775] Lustre: server umount lustre-MDT0000 complete [ 1473.398447] LustreError: 47131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184624 with bad export cookie 16143439737187596226 [ 1473.405070] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.412445] LustreError: 47131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1479.683200] Lustre: server umount lustre-MDT0001 complete [ 1489.302384] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 1491.482307] Key type lgssc unregistered [ 1491.755780] LNet: 48999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.762919] LNetError: 48999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.774911] LNet: Removed LNI 192.168.203.110@tcp [ 1492.638356] Key type .llcrypt unregistered [ 1492.639971] Key type ._llcrypt unregistered [ 1515.179703] Key type ._llcrypt registered [ 1515.185885] Key type .llcrypt registered [ 1515.281178] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1516.448767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1516.481258] alg: No test for adler32 (adler32-zlib) [ 1517.560654] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1517.760568] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 1519.473282] Key type lgssc registered [ 1520.666174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1528.241922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1528.263430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1529.614287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.894486] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1534.957438] LustreError: 50416:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.054927] LustreError: 50416:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1540.069754] LustreError: 50417:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.125596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.463163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1546.285921] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1553.586635] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1558.979599] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1565.425428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1565.683624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.817651] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1574.891533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1578.775234] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1585.099350] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1588.443862] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1600.559375] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-08 6h25m49s [ 1602.296141] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1605.610733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.626551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.646949] Lustre: Skipped 1 previous similar message [ 1605.667748] Lustre: Skipped 2 previous similar messages [ 1606.691854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.367139] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 1608.376788] Lustre: Skipped 3 previous similar messages [ 1610.543718] Lustre: server umount lustre-MDT0000 complete [ 1612.773580] LustreError: 51485:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.784042] LustreError: 51485:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1613.588585] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1618.917594] LustreError: 50417:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.931817] LustreError: 50417:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 1619.970543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.129909] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1620.363841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.457807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1621.612714] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1625.367585] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1628.832299] Lustre: DEBUG MARKER: Wait for df (36745) ... [ 1630.604626] Lustre: DEBUG MARKER: done [ 1635.809660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1635.827809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.844305] Lustre: Skipped 1 previous similar message [ 1635.860086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.834419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.426893] Lustre: server umount lustre-OST0000 complete [ 1647.084214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.127172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.152722] Lustre: Skipped 3 previous similar messages [ 1651.867576] Lustre: server umount lustre-MDT0000 complete [ 1654.769886] LustreError: 51144:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.826799] LustreError: 51144:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1655.825823] LustreError: 51151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184806 with bad export cookie 3330772375663569572 [ 1655.830391] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.832060] LustreError: 51151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1656.043611] Lustre: server umount lustre-MDT0001 complete [ 1665.579162] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 1667.793872] Key type lgssc unregistered [ 1668.098256] LNet: 55345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.123639] LNetError: 55345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.142596] LNet: Removed LNI 192.168.203.110@tcp [ 1669.840146] Key type .llcrypt unregistered [ 1669.844838] Key type ._llcrypt unregistered [ 1709.511353] Key type ._llcrypt registered [ 1709.513989] Key type .llcrypt registered [ 1709.615956] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1711.510833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.559903] alg: No test for adler32 (adler32-zlib) [ 1712.556849] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1712.770566] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 1714.432233] Key type lgssc registered [ 1716.074930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1723.221953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1723.238630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.391520] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1724.407251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1724.523700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.576845] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1737.593581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.656459] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1737.674228] Lustre: 57552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1737.839689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.312831] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1748.425454] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.538846] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.866775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.988640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1760.129766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.057543] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1770.343706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1773.093171] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.060875] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-08 6h28m50s [ 1781.723959] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1787.770039] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1789.905791] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1789.949052] LustreError: 57561:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1810.147829] Lustre: lustre-MDT0000: Client 58f53554-92ae-43b6-b068-893c07a9364a (at 192.168.203.10@tcp) reconnecting [ 1830.011056] LustreError: 57561:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1833.838426] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1836.272678] Lustre: DEBUG MARKER: done [ 1842.150212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1842.162191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.165665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.177380] Lustre: Skipped 1 previous similar message [ 1842.213953] Lustre: Skipped 1 previous similar message [ 1843.185740] Lustre: server umount lustre-OST0000 complete [ 1847.266788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.275842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.280579] Lustre: Skipped 1 previous similar message [ 1848.291295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.308391] Lustre: Skipped 3 previous similar messages [ 1853.085599] Lustre: server umount lustre-MDT0000 complete [ 1856.196668] LustreError: 57544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185006 with bad export cookie 12977662462485796755 [ 1856.199428] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1856.210267] LustreError: 57544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1856.726276] Lustre: server umount lustre-MDT0001 complete [ 1867.929455] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 1870.362341] Key type lgssc unregistered [ 1870.813712] LNet: 61883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.820205] LNetError: 61883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.840838] LNet: Removed LNI 192.168.203.110@tcp [ 1871.511846] Key type .llcrypt unregistered [ 1871.514330] Key type ._llcrypt unregistered [ 1918.215338] Key type ._llcrypt registered [ 1918.217328] Key type .llcrypt registered [ 1918.474371] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 1920.233065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.247175] alg: No test for adler32 (adler32-zlib) [ 1921.249460] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1921.651811] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 1923.475725] Key type lgssc registered [ 1924.668711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1927.625532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1927.638496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.872512] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1928.889856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1928.997575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.769923] Lustre: Failing over lustre-MDT0000 [ 1929.929475] Lustre: server umount lustre-MDT0000 complete [ 1936.702207] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:31:26 (1786185086) [ 1942.987764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.247990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.034835] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1955.108667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.244129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1955.266959] Lustre: 64681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1955.483797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1960.806204] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1969.070867] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1975.558506] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.155304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.358567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1982.634503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.390288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1989.913553] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 1998.109504] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2005.353937] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2014.183762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.213376] Lustre: Skipped 2 previous similar messages [ 2014.213810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.239212] Lustre: Skipped 2 previous similar messages [ 2018.237104] Lustre: server umount lustre-MDT0000 complete [ 2021.595193] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185172 with bad export cookie 14867456781786221761 [ 2021.598311] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.616583] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.015580] Lustre: server umount lustre-MDT0001 complete [ 2026.455970] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2030.949763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2039.771330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.736260] Lustre: 63605:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185175/real 1786185175] req@00000000c3860982 x1872950798099200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786185191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2040.749816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.848339] Lustre: 63604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185177/real 1786185177] req@00000000f5bef3cd x1872950798099328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786185193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2045.921083] Lustre: 63603:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185180/real 1786185180] req@00000000161d64a5 x1872950798099456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786185196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2046.884276] LustreError: 63602:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bfa06ff2 x1872950798100224/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.258338] LustreError: 66529:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.274836] LustreError: 66529:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 2047.339495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.416707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2048.352383] Lustre: 63605:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185182/real 1786185182] req@00000000f3051a78 x1872950798099712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786185198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2049.378756] LustreError: 66530:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.408895] LustreError: 68451:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.432378] LustreError: 68451:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2052.567673] Lustre: 63606:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185187/real 1786185187] req@000000002cc4d450 x1872950798099840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786185203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2053.343901] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2055.650931] LustreError: 66529:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.660884] LustreError: 66529:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2060.809341] LustreError: 68451:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.822820] LustreError: 68451:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2061.164498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.464053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.656751] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2066.930821] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2070.059415] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2073.255711] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2079.633417] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2086.984786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2086.992256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.999968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.654642] Lustre: server umount lustre-MDT0000 complete [ 2090.469870] LustreError: 66530:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.484747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.426537] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185242 with bad export cookie 14867456781786224036 [ 2091.430215] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.460745] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2091.872465] Lustre: server umount lustre-MDT0001 complete [ 2097.455966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2106.591484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.096149] Lustre: 63604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185246/real 1786185246] req@0000000033557338 x1872950798126848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786185262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2112.123286] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.137378] Lustre: Skipped 1 previous similar message [ 2117.282538] LustreError: 63602:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d05af777 x1872950798128000/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.619784] LustreError: 66529:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.637278] LustreError: 66529:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2117.730784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.807102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2122.229930] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2130.417564] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2130.545131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.505455] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2136.049413] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2143.300424] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.227099] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.524043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.525881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2156.531604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.557419] Lustre: Skipped 1 previous similar message [ 2161.634712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.644084] Lustre: Skipped 4 previous similar messages [ 2162.487063] Lustre: server umount lustre-MDT0000 complete [ 2166.757979] LustreError: 69185:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.781897] LustreError: 69185:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 8 previous similar messages [ 2166.998129] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185317 with bad export cookie 14867456781786225415 [ 2167.008517] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.028671] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2167.277920] Lustre: server umount lustre-MDT0001 complete [ 2173.095559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2176.462369] Lustre: DEBUG MARKER: files compared the same [ 2187.466620] Lustre: server umount lustre-OST0000 complete [ 2204.164373] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 2207.066432] Key type lgssc unregistered [ 2207.360606] LNet: 74410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.366633] LNetError: 74410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.377352] LNet: Removed LNI 192.168.203.110@tcp [ 2208.170327] Key type .llcrypt unregistered [ 2208.172683] Key type ._llcrypt unregistered [ 2229.400898] Key type ._llcrypt registered [ 2229.405400] Key type .llcrypt registered [ 2229.544807] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2230.736885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.887436] alg: No test for adler32 (adler32-zlib) [ 2231.983772] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2232.293256] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 2234.140761] Key type lgssc registered [ 2235.242439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.658220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2241.671585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.146358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.555627] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug +malloc all [ 2253.282498] LustreError: 75826:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.304297] LustreError: 75826:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2254.306324] LustreError: 75827:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.520211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.818537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.580677] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug +malloc all [ 2267.665324] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2272.626819] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2278.055174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2278.281342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.696399] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug +malloc all [ 2288.491264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2289.986900] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2298.850519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.851844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.853254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.880281] Lustre: Skipped 1 previous similar message [ 2303.093801] Lustre: server umount lustre-OST0000 complete [ 2307.041874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.046678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.053989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.057574] Lustre: Skipped 1 previous similar message [ 2309.088792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.089994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.344600] Lustre: server umount lustre-MDT0000 complete [ 2314.211134] LustreError: 75821:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.243937] LustreError: 75821:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2315.062771] LustreError: 75808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185465 with bad export cookie 14231839486551216342 [ 2315.066417] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.075970] LustreError: 75808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2315.453141] Lustre: server umount lustre-MDT0001 complete [ 2324.394538] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 2326.584964] Key type lgssc unregistered [ 2326.800604] LNet: 79360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.811681] LNetError: 79360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.823875] LNet: Removed LNI 192.168.203.110@tcp [ 2327.475593] Key type .llcrypt unregistered [ 2327.480160] Key type ._llcrypt unregistered [ 2346.782552] Key type ._llcrypt registered [ 2346.785254] Key type .llcrypt registered [ 2346.883374] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2347.636092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.683195] alg: No test for adler32 (adler32-zlib) [ 2348.730379] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2348.903838] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 2350.568355] Key type lgssc registered [ 2351.644802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2359.260307] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2359.277861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2374.752593] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2379.872416] LustreError: 80728:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.110@tcp: failed processing log, type 4: rc = -110 [ 2406.560210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.124693] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2419.714955] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.405767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.549201] Lustre: *** cfs_fail_loc=706, val=0*** [ 2428.063997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.141758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2432.389988] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2433.506681] LustreError: 80755:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.519769] LustreError: 80755:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 2438.634068] LustreError: 80754:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.726016] LustreError: 82509:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.759014] LustreError: 82509:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2442.511059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.768350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2446.720877] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2452.903600] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2458.426472] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2463.222126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2463.222632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.247887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.251780] Lustre: Skipped 1 previous similar message [ 2466.524608] Lustre: server umount lustre-OST0000 complete [ 2473.445591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.446688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2473.449972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.449980] Lustre: Skipped 1 previous similar message [ 2473.459379] Lustre: Skipped 1 previous similar message [ 2476.431127] Lustre: server umount lustre-MDT0000 complete [ 2478.561630] LustreError: 81819:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.602205] LustreError: 81819:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2479.630180] LustreError: 80735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185630 with bad export cookie 17002445226122690760 [ 2479.633683] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.642768] LustreError: 80735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2479.935377] Lustre: server umount lustre-MDT0001 complete [ 2490.816520] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 2493.436253] Key type lgssc unregistered [ 2493.685594] LNet: 84347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.696960] LNetError: 84347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.723947] LNet: Removed LNI 192.168.203.110@tcp [ 2494.434339] Key type .llcrypt unregistered [ 2494.437929] Key type ._llcrypt unregistered [ 2518.353577] Key type ._llcrypt registered [ 2518.355603] Key type .llcrypt registered [ 2518.496197] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2520.072896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2520.303784] alg: No test for adler32 (adler32-zlib) [ 2521.580676] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2521.749500] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 2523.458822] Key type lgssc registered [ 2524.744100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2532.447188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2532.459300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.100996] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2545.668143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.423967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2550.791478] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2551.778819] LustreError: 86453:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.804559] LustreError: 86453:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 2556.901032] LustreError: 86452:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.913978] LustreError: 86452:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2559.109369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2559.321193] LustreError: 86456:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.349556] LustreError: 86456:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2559.368261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.584503] LustreError: 86457:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.598443] LustreError: 86457:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2565.401351] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2569.724027] LustreError: 86452:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.735290] LustreError: 86452:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2573.124762] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.136170] Lustre: MGS: Client 1496e7bf-e02e-44c4-a9dd-8d277af3b5cd (at 0@lo) reconnecting [ 2573.154027] LustreError: 86467:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2573.338207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.387209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2577.580663] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2593.779974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.817689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.823218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2596.978671] Lustre: server umount lustre-OST0000 complete [ 2604.003737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.004866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.010388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.010398] Lustre: Skipped 1 previous similar message [ 2604.063069] Lustre: Skipped 2 previous similar messages [ 2606.181273] Lustre: server umount lustre-MDT0000 complete [ 2607.014301] LustreError: 86457:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.069736] LustreError: 86457:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 2610.569851] Lustre: server umount lustre-MDT0001 complete [ 2614.474626] Lustre: server umount MGS complete [ 2625.343557] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:42:55 (1786185775) [ 2652.013703] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 2660.728323] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2693.945789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.200872] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2707.821436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.010213] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2708.044977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2708.104421] Lustre: lustre-MDT0001: new disk, initializing [ 2708.161707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2708.193985] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2708.206127] Lustre: 93618:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2708.227439] LustreError: 93618:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2712.834539] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2723.192056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.412538] Lustre: lustre-OST0000: new disk, initializing [ 2723.416297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2723.421281] Lustre: 94385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2723.508830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.272302] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2731.747729] Lustre: 94804:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000554ca717 x1872951427243648/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786185898 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2739.074329] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2745.408700] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.418839] Lustre: MGS: Client fa88c0b1-90c7-4095-8582-4115a66a7610 (at 0@lo) reconnecting [ 2745.462660] Lustre: 92887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2745.485651] LustreError: 93604:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2745.497602] LustreError: 93604:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2745.521500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2745.679901] Lustre: lustre-MDT0000: new disk, initializing [ 2746.106865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.185505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2746.301565] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2747.237553] Lustre: 95453:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000075ffae4f x1872951427250944/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786185913 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2750.740457] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2752.305364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2752.315348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2752.455065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2777.063396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.073919] Lustre: Skipped 1 previous similar message [ 2777.075271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.089212] Lustre: Skipped 2 previous similar messages [ 2782.178515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2782.191701] Lustre: Skipped 1 previous similar message [ 2787.301088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2787.304982] Lustre: Skipped 1 previous similar message [ 2788.832144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2788.990037] Lustre: server umount lustre-OST0000 complete [ 2792.419035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.429855] Lustre: Skipped 1 previous similar message [ 2797.544299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2797.562722] Lustre: Skipped 3 previous similar messages [ 2798.605757] Lustre: server umount lustre-MDT0000 complete [ 2808.386165] Lustre: server umount lustre-MDT0001 complete [ 2811.899924] Lustre: server umount MGS complete [ 2825.650594] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:46:15 (1786185975) [ 2843.248721] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 2845.299390] Key type lgssc unregistered [ 2845.539056] LNet: 97957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2845.551576] LNetError: 97957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2846.630632] LNet: Removed LNI 192.168.203.110@tcp [ 2847.352983] Key type .llcrypt unregistered [ 2847.355531] Key type ._llcrypt unregistered [ 2863.520500] Key type ._llcrypt registered [ 2863.525635] Key type .llcrypt registered [ 2863.623325] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2864.498922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.597347] alg: No test for adler32 (adler32-zlib) [ 2865.794382] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2866.107416] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 2867.848230] Key type lgssc registered [ 2869.472468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2877.204625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2877.218821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.877612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.376872] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2889.198307] LustreError: 99181:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.216175] LustreError: 99181:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2889.298083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.500752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.632711] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2900.452485] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2905.460163] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2911.827341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2912.096930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.630758] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 2923.461182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2927.680453] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2938.851496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.851608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.857110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.883671] Lustre: Skipped 1 previous similar message [ 2943.539919] Lustre: server umount lustre-OST0000 complete [ 2949.096695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.104260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.115744] Lustre: Skipped 1 previous similar message [ 2949.140372] Lustre: Skipped 2 previous similar messages [ 2954.210139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.225316] Lustre: Skipped 1 previous similar message [ 2958.370223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2958.374442] Lustre: Skipped 1 previous similar message [ 2961.393196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2961.555503] Lustre: server umount lustre-MDT0000 complete [ 2963.426467] LustreError: 99176:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.439990] LustreError: 99176:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2965.887806] LustreError: 99161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186116 with bad export cookie 17793285849581564648 [ 2965.891614] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.897288] LustreError: 99161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2966.232840] Lustre: server umount lustre-MDT0001 complete [ 2976.613508] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 2978.543897] Key type lgssc unregistered [ 2978.874459] LNet: 102718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.885833] LNetError: 102718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2978.913399] LNet: Removed LNI 192.168.203.110@tcp [ 2979.587212] Key type .llcrypt unregistered [ 2979.588984] Key type ._llcrypt unregistered [ 2995.108497] Key type ._llcrypt registered [ 2995.110659] Key type .llcrypt registered [ 2995.199958] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 2996.728727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2996.881971] alg: No test for adler32 (adler32-zlib) [ 2998.075226] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2998.369432] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3000.070787] Key type lgssc registered [ 3001.014498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3006.507474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3006.518378] LustreError: 103790:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3006.596184] LustreError: 103801:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3006.601899] LustreError: 103790:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4925 [ 3006.612817] LustreError: 103790:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3006.623182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.623777] LustreError: 103790:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3007.984987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.517571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.705586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.614915] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3024.776742] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing lsmod [ 3028.553376] LustreError: 105004:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3028.697852] LustreError: 105008:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3028.702350] LustreError: 105004:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4874 [ 3028.738282] LustreError: 105004:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3028.752959] LustreError: 105004:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3028.804014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.503672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.595316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.602412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.614884] Lustre: Skipped 2 previous similar messages [ 3034.630345] Lustre: Skipped 1 previous similar message [ 3038.463405] Lustre: server umount lustre-MDT0000 complete [ 3039.713922] LustreError: 103839:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.728567] LustreError: 103839:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 3040.748321] LustreError: 103835:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.777199] LustreError: 103835:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 3042.587730] LustreError: 103815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186193 with bad export cookie 12582493029465868785 [ 3042.602942] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3042.864355] LustreError: 104270:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006110cabc x1872951927393536/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3043.049768] Lustre: server umount lustre-MDT0001 complete [ 3055.591603] Lustre: server umount lustre-OST0000 complete [ 3063.266958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.632862] LustreError: 106070:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.689848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.138477] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3068.948081] LustreError: 106071:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.024062] LustreError: 106070:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.682962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.972835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.222806] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3088.636335] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.760466] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3099.906881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.118879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.330285] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3109.376689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3113.747416] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3123.075202] Lustre: server umount lustre-OST0000 complete [ 3128.290956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3128.305115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.328812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.831308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.843566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.863315] Lustre: Skipped 1 previous similar message [ 3134.947375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.951027] Lustre: Skipped 1 previous similar message [ 3137.051168] Lustre: server umount lustre-MDT0000 complete [ 3139.785985] LustreError: 106052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186290 with bad export cookie 12582493029465869471 [ 3139.799273] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.803037] LustreError: 106052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3149.438832] Lustre: server umount lustre-MDT0001 complete [ 3160.946938] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 3163.414342] Key type lgssc unregistered [ 3163.787695] LNet: 109609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3163.793181] LNetError: 109609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.803572] LNet: Removed LNI 192.168.203.110@tcp [ 3164.533835] Key type .llcrypt unregistered [ 3164.536254] Key type ._llcrypt unregistered [ 3188.254318] Key type ._llcrypt registered [ 3188.260418] Key type .llcrypt registered [ 3188.351941] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3191.290188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3191.310044] alg: No test for adler32 (adler32-zlib) [ 3192.376942] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3192.570698] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3194.200177] Key type lgssc registered [ 3195.521397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3203.339747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3203.367890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.845206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.891423] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3215.330802] LustreError: 111005:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.352757] LustreError: 111005:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 3218.923636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.192735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3223.988385] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3231.915402] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3237.188921] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3244.565781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3244.799652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3250.340178] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3250.954742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3257.423202] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3266.357079] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3269.359761] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3274.737871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.762266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3274.763226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3274.784260] Lustre: Skipped 1 previous similar message [ 3278.749543] Lustre: server umount lustre-OST0000 complete [ 3284.966018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.968050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.988054] Lustre: Skipped 2 previous similar messages [ 3284.997378] Lustre: Skipped 1 previous similar message [ 3288.478531] Lustre: server umount lustre-MDT0000 complete [ 3290.082327] LustreError: 111004:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.089462] LustreError: 111004:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3292.223543] LustreError: 110985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186442 with bad export cookie 615299365564533537 [ 3292.235948] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.236256] LustreError: 110985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3292.682191] Lustre: server umount lustre-MDT0001 complete [ 3304.012263] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 3307.172346] Key type lgssc unregistered [ 3307.725952] LNet: 114649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3307.734477] LNetError: 114649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3307.850477] LNet: Removed LNI 192.168.203.110@tcp [ 3309.778457] Key type .llcrypt unregistered [ 3309.781802] Key type ._llcrypt unregistered [ 3331.757231] Key type ._llcrypt registered [ 3331.761719] Key type .llcrypt registered [ 3331.980740] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3333.417219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.437246] alg: No test for adler32 (adler32-zlib) [ 3335.145184] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3335.359190] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3337.104224] Key type lgssc registered [ 3339.132896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3348.397533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3348.414715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.787136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.133580] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3360.232177] LustreError: 115873:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.240919] LustreError: 115873:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 3362.051520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.402544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.719221] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3375.462610] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3381.241853] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3388.819846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.247531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.368535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3395.062891] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3401.853868] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3415.010455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.011435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.022081] LustreError: Skipped 1 previous similar message [ 3415.034245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.058392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.060221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.065561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.078367] Lustre: Skipped 1 previous similar message [ 3419.500570] Lustre: server umount lustre-OST0000 complete [ 3423.715569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3423.724616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.732970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.299669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.311671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.973116] Lustre: server umount lustre-MDT0000 complete [ 3432.419956] LustreError: 115866:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.446883] LustreError: 115866:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3434.005197] LustreError: 115853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186584 with bad export cookie 608423943079042136 [ 3434.014547] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.018554] LustreError: 115853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3434.308560] Lustre: server umount lustre-MDT0001 complete [ 3444.854603] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 3447.646366] Key type lgssc unregistered [ 3447.919385] LNet: 119410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3447.932885] LNetError: 119410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3448.998448] LNet: Removed LNI 192.168.203.110@tcp [ 3449.832963] Key type .llcrypt unregistered [ 3449.835814] Key type ._llcrypt unregistered [ 3475.567781] Key type ._llcrypt registered [ 3475.569861] Key type .llcrypt registered [ 3475.672921] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3477.464194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3477.473871] alg: No test for adler32 (adler32-zlib) [ 3478.434578] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3478.675316] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3480.336453] Key type lgssc registered [ 3481.145329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.167680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.178968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.824585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.734639] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3501.028955] LustreError: 120829:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.045195] LustreError: 120829:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 3505.314495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.633735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.188272] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3515.842547] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3520.777332] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3526.756548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.041449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.386793] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3536.337501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3541.494897] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3548.550434] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3552.082589] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3552.086652] Lustre: Skipped 2 previous similar messages [ 3555.384771] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3555.415698] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3556.797312] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3556.800764] Lustre: Skipped 2 previous similar messages [ 3567.205834] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3567.212805] Lustre: Skipped 1 previous similar message [ 3574.078697] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.10@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3574.086194] Lustre: Skipped 2 previous similar messages [ 3574.859844] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.10@tcp 192.168.0.[2,10]@tcp [ 3584.115909] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.10/24@tcp in log lustre-MDT0000 [ 3584.189920] Lustre: Skipped 2 previous similar messages [ 3593.786440] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.10/24@tcp [ 3593.794964] Lustre: Skipped 1 previous similar message [ 3599.856780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3599.858097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.867834] LustreError: Skipped 2 previous similar messages [ 3599.896157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.631882] Lustre: server umount lustre-OST0000 complete [ 3608.035065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.035966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3608.043495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.047121] LustreError: Skipped 1 previous similar message [ 3610.515154] Lustre: server umount lustre-MDT0000 complete [ 3613.154372] LustreError: 120829:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.200103] LustreError: 120829:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3613.617800] LustreError: 120809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186764 with bad export cookie 13498849884552975717 [ 3613.626768] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3613.628206] LustreError: 120809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3624.019679] Lustre: server umount lustre-MDT0001 complete [ 3634.623969] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 3637.934905] Key type lgssc unregistered [ 3638.215043] LNet: 125926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3638.220766] LNetError: 125926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3638.240640] LNet: Removed LNI 192.168.203.110@tcp [ 3639.481150] Key type .llcrypt unregistered [ 3639.484319] Key type ._llcrypt unregistered [ 3667.580869] Key type ._llcrypt registered [ 3667.591699] Key type .llcrypt registered [ 3667.721450] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3669.475645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.844115] alg: No test for adler32 (adler32-zlib) [ 3671.523391] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3671.762629] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3673.408257] Key type lgssc registered [ 3674.286614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3682.699835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3682.711569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.133962] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3684.177946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.181789] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.10/24@tcp [ 3688.859834] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3693.578694] LustreError: 127485:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.609890] LustreError: 127485:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 3698.258646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.473375] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3698.579310] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.10/24@tcp [ 3698.690639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3703.154223] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3710.699501] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3716.353211] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3723.236294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3723.584067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.710687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3729.423129] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3737.931575] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3749.346743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3749.349500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3749.351972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3749.356375] Lustre: Skipped 1 previous similar message [ 3753.132270] Lustre: server umount lustre-OST0000 complete [ 3759.587220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.596824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.606872] Lustre: Skipped 1 previous similar message [ 3763.855993] Lustre: server umount lustre-MDT0000 complete [ 3764.707705] LustreError: 127486:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3764.727320] LustreError: 127486:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 3767.280061] LustreError: 127465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186917 with bad export cookie 6178257842605463148 [ 3767.290173] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3767.297695] LustreError: 127465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3767.820421] Lustre: server umount lustre-MDT0001 complete [ 3779.164634] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 3781.800218] Key type lgssc unregistered [ 3782.001664] LNet: 131087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3782.012422] LNetError: 131087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3782.027045] LNet: Removed LNI 192.168.203.110@tcp [ 3782.818669] Key type .llcrypt unregistered [ 3782.823109] Key type ._llcrypt unregistered [ 3809.887196] Key type ._llcrypt registered [ 3809.889326] Key type .llcrypt registered [ 3809.981611] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 3824.806571] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 3826.049872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3826.080461] alg: No test for adler32 (adler32-zlib) [ 3827.123898] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 3827.292137] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 3829.056192] Key type lgssc registered [ 3830.372851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3863.701498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3863.724418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.947225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3864.970689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3865.017684] Lustre: lustre-MDT0000: new disk, initializing [ 3865.069176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.083793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3870.414775] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3881.313219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.405260] Lustre: 134491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3881.450928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3881.454092] Lustre: Skipped 1 previous similar message [ 3881.534245] Lustre: lustre-MDT0001: new disk, initializing [ 3881.644539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3881.676867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3881.689441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3886.370920] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3895.111218] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3900.662157] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3908.536829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3908.962532] Lustre: lustre-OST0000: new disk, initializing [ 3908.981842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3908.996281] Lustre: 136458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3909.223817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.474300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3915.491872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3915.725603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3916.223951] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3926.586843] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3931.369655] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3931.634358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3936.845606] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3937.559611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3939.315911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.320814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.324859] Lustre: Skipped 1 previous similar message [ 3939.333450] Lustre: Skipped 1 previous similar message [ 3944.418121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3944.424294] Lustre: Skipped 1 previous similar message [ 3945.275744] Lustre: server umount lustre-OST0000 complete [ 3949.538133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.553296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.556840] Lustre: Skipped 1 previous similar message [ 3949.565658] Lustre: Skipped 1 previous similar message [ 3954.659462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3954.673517] Lustre: Skipped 1 previous similar message [ 3955.432365] Lustre: server umount lustre-MDT0000 complete [ 3959.448336] LustreError: 134481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187110 with bad export cookie 3644057156927027879 [ 3959.458313] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3959.780878] Lustre: server umount lustre-MDT0001 complete [ 3968.801950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.140916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.863733] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3974.635283] LustreError: 138676:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.646649] LustreError: 138676:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 3978.722466] LustreError: 138677:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.361949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.633287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3988.600509] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 3997.108460] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4002.624421] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4009.174062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4009.420931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.394255] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4021.973722] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4026.497724] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4026.797935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4030.877973] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4031.038818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4040.483908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4040.701541] Lustre: lustre-OST0001: new disk, initializing [ 4040.707288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4040.719147] Lustre: 142047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4040.825996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.696863] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4048.495917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4048.525659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4048.762963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4056.923499] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4057.093221] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4062.065187] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4062.267380] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4067.407254] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c5bc5e27000.ost_server_uuid 50 [ 4069.223343] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c5bc5e27000.ost_server_uuid in IDLE state after 0 sec [ 4079.586187] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.586648] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.594729] LustreError: Skipped 1 previous similar message [ 4079.602119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4079.624658] Lustre: Skipped 1 previous similar message [ 4082.278205] Lustre: server umount lustre-OST0001 complete [ 4084.729217] LustreError: 140498:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.738255] LustreError: 140498:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 4089.850346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.855724] LustreError: 142435:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.866411] LustreError: 142435:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 4089.880437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.884717] Lustre: Skipped 2 previous similar messages [ 4092.590381] Lustre: server umount lustre-OST0000 complete [ 4100.068244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.101656] Lustre: Skipped 2 previous similar messages [ 4102.986745] Lustre: server umount lustre-MDT0000 complete [ 4105.192163] LustreError: 138677:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.222427] LustreError: 138677:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 4107.207069] LustreError: 138658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187257 with bad export cookie 3644057156927028754 [ 4107.207564] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4107.224736] LustreError: 138658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4107.495382] Lustre: server umount lustre-MDT0001 complete [ 4126.766163] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 4129.038893] Key type lgssc unregistered [ 4129.238745] LNet: 145054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4129.246157] LNetError: 145054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4129.256860] LNet: Removed LNI 192.168.203.110@tcp [ 4130.037395] Key type .llcrypt unregistered [ 4130.039913] Key type ._llcrypt unregistered [ 4169.337684] Key type ._llcrypt registered [ 4169.341637] Key type .llcrypt registered [ 4169.477297] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 4184.780139] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 4186.984623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4187.035078] alg: No test for adler32 (adler32-zlib) [ 4188.721495] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 4189.063077] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 4190.944171] Key type lgssc registered [ 4191.740919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4225.371307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4225.403248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.803331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4226.836517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4226.912206] Lustre: lustre-MDT0000: new disk, initializing [ 4226.988045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.003956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4232.597752] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4242.735165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.861564] Lustre: 149218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4242.947477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4242.956539] Lustre: Skipped 1 previous similar message [ 4243.430082] Lustre: lustre-MDT0001: new disk, initializing [ 4243.574464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.727654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4243.744707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4249.096664] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4260.266739] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4266.041828] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4272.353373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4272.590966] Lustre: lustre-OST0000: new disk, initializing [ 4272.599704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4272.608776] Lustre: 151184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4272.695276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.070830] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4282.974885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4283.001858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4283.224025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4288.237737] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4293.580303] Lustre: Failing over lustre-OST0000 [ 4293.604565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.605297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.614521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.621556] LustreError: Skipped 1 previous similar message [ 4293.790612] Lustre: Skipped 1 previous similar message [ 4293.962698] Lustre: server umount lustre-OST0000 complete [ 4314.430446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.648265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.658037] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4321.588859] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4323.120446] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4325.073033] Lustre: Failing over lustre-MDT0000 [ 4325.265909] Lustre: server umount lustre-MDT0000 complete [ 4325.350094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4325.394481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.955632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.982888] Lustre: Skipped 1 previous similar message [ 4333.759459] LustreError: 149222:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4333.796579] LustreError: 149222:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 6 previous similar messages [ 4338.864767] LustreError: 149222:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4338.907983] LustreError: 149222:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 4343.308295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.394264] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.637268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4343.672759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.745190] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4343.997157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4347.870314] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4348.917510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4348.938117] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4348.958400] Lustre: Skipped 1 previous similar message [ 4349.091739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4359.144687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4359.147682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.161230] LustreError: Skipped 1 previous similar message [ 4359.176082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.194676] Lustre: Skipped 1 previous similar message [ 4364.262217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4364.275639] Lustre: Skipped 2 previous similar messages [ 4369.413626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4369.419011] Lustre: Skipped 1 previous similar message [ 4370.400131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4370.539933] Lustre: server umount lustre-OST0000 complete [ 4374.497966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4374.499257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.505893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4374.516168] Lustre: Skipped 2 previous similar messages [ 4380.534159] Lustre: server umount lustre-MDT0000 complete [ 4383.975561] LustreError: 149209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187534 with bad export cookie 15153909200150816004 [ 4383.984600] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4383.992925] LustreError: 149209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4384.350491] Lustre: server umount lustre-MDT0001 complete [ 4395.277534] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 4397.634395] Key type lgssc unregistered [ 4397.913701] LNet: 154840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4397.920296] LNetError: 154840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4397.929861] LNet: Removed LNI 192.168.203.110@tcp [ 4398.782382] Key type .llcrypt unregistered [ 4398.788573] Key type ._llcrypt unregistered [ 4438.371864] Key type ._llcrypt registered [ 4438.374088] Key type .llcrypt registered [ 4438.606355] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 4440.383619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4440.431942] alg: No test for adler32 (adler32-zlib) [ 4441.411417] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 4441.677913] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 4443.336444] Key type lgssc registered [ 4444.714490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4452.930492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4452.946745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.375535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.200373] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4464.642412] LustreError: 156361:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.675590] LustreError: 156361:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 4468.455842] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4469.733368] LustreError: 156362:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.259203] LustreError: 156390:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4472.270523] LustreError: 156390:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4472.275760] LustreError: 156390:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4472.446941] Lustre: server umount lustre-MDT0000 complete [ 4488.726177] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 4498.206766] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 4537.913338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.156629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4538.188422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4538.246730] Lustre: lustre-MDT0000: new disk, initializing [ 4538.294588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.304554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4543.992140] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4554.187589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.304816] Lustre: 159698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4554.350966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4554.354557] Lustre: Skipped 1 previous similar message [ 4554.408943] Lustre: lustre-MDT0001: new disk, initializing [ 4554.588647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4554.693053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4554.710103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4560.260376] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4570.594722] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4577.732160] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4583.979987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.319610] Lustre: lustre-OST0000: new disk, initializing [ 4584.323223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4584.329297] Lustre: 161666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4584.412810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.232954] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4594.306135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4594.340324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4594.463613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4599.996627] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4606.097441] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4608.864353] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:15:57 (1786187757) [ 4840.596709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.631617] Lustre: Skipped 1 previous similar message [ 4840.688683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4840.969512] Lustre: Skipped 2 previous similar messages [ 4845.968592] Lustre: server umount lustre-MDT0000 complete [ 4846.050858] LustreError: 159708:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.065755] LustreError: 159708:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 4850.662371] LustreError: 159689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188001 with bad export cookie 9090843190397105626 [ 4850.665822] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.674952] LustreError: 159689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4851.169131] LustreError: 159704:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.171361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.215546] Lustre: Skipped 1 previous similar message [ 4851.220766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4855.200684] LustreError: 159703:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.201121] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4857.211270] Lustre: server umount lustre-MDT0001 complete [ 4867.537792] Lustre: server umount lustre-OST0000 complete [ 4877.903409] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 4889.059724] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 4929.083682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.394348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4929.441330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4929.561628] Lustre: lustre-MDT0000: new disk, initializing [ 4929.648936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4929.679838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4934.568142] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4945.845573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.075822] Lustre: 166325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4946.212405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4946.219757] Lustre: Skipped 1 previous similar message [ 4946.410563] Lustre: lustre-MDT0001: new disk, initializing [ 4946.511357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4946.547547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4946.574955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4950.389787] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4960.843706] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4967.795275] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4976.021645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4976.265983] Lustre: lustre-OST0000: new disk, initializing [ 4976.269639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4976.276764] Lustre: 168292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4976.379640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.603736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4983.678907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4984.125987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4984.747546] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 4997.733732] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5003.023953] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5003.355684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5008.704385] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5008.922896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5014.498167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5014.502449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.510550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5015.023618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5017.707741] Lustre: server umount lustre-OST0000 complete [ 5023.204678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5023.209177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.215942] Lustre: Skipped 1 previous similar message [ 5023.223839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5023.231804] Lustre: Skipped 2 previous similar messages [ 5027.769745] Lustre: server umount lustre-MDT0000 complete [ 5030.375834] LustreError: 167760:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5030.392881] LustreError: 167760:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5032.494773] LustreError: 169632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188183 with bad export cookie 9090843190397264456 [ 5032.505612] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5032.507352] LustreError: 169632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5032.747584] Lustre: server umount lustre-MDT0001 complete [ 5043.551912] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:23:13 (1786188193) [ 5060.557643] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 5073.399197] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5115.875962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.205869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5116.231918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5116.299738] Lustre: lustre-MDT0000: new disk, initializing [ 5116.388153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.413664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5121.539656] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5134.452619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.553723] Lustre: 173050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5134.806784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5139.007348] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5149.819903] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5157.609804] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5166.382246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5166.601971] Lustre: lustre-OST0000: new disk, initializing [ 5166.604673] Lustre: Skipped 1 previous similar message [ 5166.607861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5166.613191] Lustre: Skipped 2 previous similar messages [ 5166.618489] Lustre: 175028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5172.135597] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5175.342334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5175.349932] Lustre: Skipped 1 previous similar message [ 5175.354452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5175.524918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5181.695941] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5191.142252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5191.145679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.157249] LustreError: Skipped 1 previous similar message [ 5191.160810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5191.160818] Lustre: Skipped 2 previous similar messages [ 5191.186655] Lustre: Skipped 2 previous similar messages [ 5195.480140] Lustre: server umount lustre-OST0000 complete [ 5201.385984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5201.407756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.519915] Lustre: Skipped 1 previous similar message [ 5210.593801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5210.602332] Lustre: Skipped 6 previous similar messages [ 5215.200135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5215.355898] Lustre: server umount lustre-MDT0000 complete [ 5215.718106] LustreError: 173062:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.745350] LustreError: 173062:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5219.521529] LustreError: 173041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188370 with bad export cookie 9090843190397265338 [ 5219.546786] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.550167] LustreError: 173041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5219.815845] Lustre: server umount lustre-MDT0001 complete [ 5231.046323] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:26:21 (1786188381) [ 5249.869320] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 5261.156099] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5306.417023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.723677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5306.753322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5306.867949] Lustre: lustre-MDT0000: new disk, initializing [ 5306.947445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.954946] Lustre: Skipped 2 previous similar messages [ 5306.965808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5310.767606] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5323.534324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.785941] Lustre: 179355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5324.105740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5329.031452] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5338.724854] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.534885] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5351.536340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5351.795070] Lustre: 181330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5359.088647] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5360.268295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5360.388135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5370.213348] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5381.090270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5381.091202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.126193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.132335] Lustre: Skipped 1 previous similar message [ 5383.456640] Lustre: server umount lustre-OST0000 complete [ 5395.422322] Lustre: server umount lustre-MDT0000 complete [ 5396.456345] LustreError: 179366:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.464753] LustreError: 179366:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5399.242258] LustreError: 179348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188549 with bad export cookie 9090843190397266423 [ 5399.252940] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5399.260776] LustreError: 179348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5399.607404] Lustre: server umount lustre-MDT0001 complete [ 5411.120799] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 5414.456634] Key type lgssc unregistered [ 5414.695738] LNet: 183194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5414.699623] LNetError: 183194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5415.719691] LNet: Removed LNI 192.168.203.110@tcp [ 5416.672150] Key type .llcrypt unregistered [ 5416.674418] Key type ._llcrypt unregistered [ 5445.161478] Key type ._llcrypt registered [ 5445.164987] Key type .llcrypt registered [ 5445.291920] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5447.131046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5447.442488] alg: No test for adler32 (adler32-zlib) [ 5449.293903] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5449.602769] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 5451.304157] Key type lgssc registered [ 5452.572258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5460.879153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5460.893190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.287210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.704117] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5471.730082] LustreError: 184609:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5471.800548] LustreError: 184609:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5475.826570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.052231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5480.137129] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5487.651686] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5492.566700] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5500.244317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5500.613324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.944895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5507.734376] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5516.214771] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5537.766666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.769887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5537.799949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5538.017899] Lustre: server umount lustre-OST0000 complete [ 5542.883184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.885549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5542.899798] Lustre: Skipped 1 previous similar message [ 5542.907037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5542.940927] Lustre: Skipped 2 previous similar messages [ 5548.002547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5548.017867] Lustre: Skipped 1 previous similar message [ 5548.176641] Lustre: server umount lustre-MDT0000 complete [ 5551.014740] LustreError: 184610:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.047598] LustreError: 184610:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5552.671416] LustreError: 184589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188703 with bad export cookie 10611923655424771484 [ 5552.674240] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5552.682039] LustreError: 184589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5552.979758] Lustre: server umount lustre-MDT0001 complete [ 5564.031798] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 5566.237911] Key type lgssc unregistered [ 5566.484947] LNet: 188168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5566.494248] LNetError: 188168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5566.516367] LNet: Removed LNI 192.168.203.110@tcp [ 5567.110157] Key type .llcrypt unregistered [ 5567.116078] Key type ._llcrypt unregistered [ 5590.753706] Key type ._llcrypt registered [ 5590.760721] Key type .llcrypt registered [ 5590.956268] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 5592.818267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5592.981539] alg: No test for adler32 (adler32-zlib) [ 5594.004895] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 5594.378640] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 5596.178112] Key type lgssc registered [ 5597.546950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5605.732925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5605.756037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.106287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.135552] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5616.612637] LustreError: 189565:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.628278] LustreError: 189565:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 5617.789236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.131686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5622.293744] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5629.736751] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5635.641364] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5644.643613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5644.899669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.882921] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5652.075923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5659.016610] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5672.127503] Lustre: lustre-OST0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 5672.930289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5672.942391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.957163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.972656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.989264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.996094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.232826] Lustre: lustre-OST0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 5677.249430] Lustre: Skipped 1 previous similar message [ 5682.358975] Lustre: lustre-OST0000: Not available for connect from 192.168.203.10@tcp (stopping) [ 5682.364898] Lustre: Skipped 2 previous similar messages [ 5684.192196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5684.293350] Lustre: server umount lustre-OST0000 complete [ 5691.519374] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c5bc5e23000.ost_server_uuid 50 [ 5693.299270] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5709.795667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.816387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.822066] Lustre: Skipped 3 previous similar messages [ 5713.051199] Lustre: server umount lustre-MDT0000 complete [ 5714.924018] LustreError: 189561:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.961914] LustreError: 189561:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5715.946804] LustreError: 190293:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5715.971782] LustreError: 190293:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5717.276911] LustreError: 189546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188867 with bad export cookie 10210355351751177505 [ 5717.311875] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5717.322488] LustreError: 189546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5717.782874] Lustre: server umount lustre-MDT0001 complete [ 5728.134197] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:34:37 (1786188877) [ 5735.427894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.766415] LustreError: 193481:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.832843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.030777] LustreError: 193482:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.625395] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5751.282936] LustreError: 193482:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.320669] LustreError: 193482:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5752.003854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5752.392385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5757.317417] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5765.446446] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5773.606544] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5780.530261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5780.917349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.967167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5786.702891] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5793.685212] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5799.930285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.193215] Lustre: lustre-OST0001: new disk, initializing [ 5800.196936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5800.204615] Lustre: 196322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5800.311866] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5806.145884] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5809.238302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5809.249422] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5809.316755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5814.439672] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5817.825541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5817.825860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.839478] LustreError: Skipped 1 previous similar message [ 5817.841219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5817.848781] Lustre: Skipped 2 previous similar messages [ 5817.861790] Lustre: Skipped 1 previous similar message [ 5823.384879] Lustre: server umount lustre-OST0000 complete [ 5824.998887] LustreError: 195303:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.045601] LustreError: 195303:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 5830.151325] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5830.369872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5834.753797] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5835.115627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5848.553852] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.574820] Lustre: Skipped 1 previous similar message [ 5853.675548] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5853.680567] Lustre: Skipped 6 previous similar messages [ 5854.367295] Lustre: server umount lustre-OST0001 complete [ 5858.788920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.805398] Lustre: Skipped 1 previous similar message [ 5864.794685] Lustre: server umount lustre-MDT0000 complete [ 5868.510620] LustreError: 193463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189019 with bad export cookie 10210355351751178576 [ 5868.513895] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.535613] LustreError: 193463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5868.893462] Lustre: server umount lustre-MDT0001 complete [ 5891.221792] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:37:21 (1786189041) [ 5897.727702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.993436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5898.025996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5898.185881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.659577] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5911.165286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.244557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5911.272398] Lustre: 199759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5916.196318] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5923.544491] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5931.894052] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5939.980174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5940.234262] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5946.420656] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5949.036844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5954.143973] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5961.487127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5961.535815] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5968.622494] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 5976.519240] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5979.617343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5979.621689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.629507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5979.633428] Lustre: Skipped 4 previous similar messages [ 5985.476984] Lustre: server umount lustre-OST0000 complete [ 5988.014038] LustreError: 201609:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.067323] LustreError: 201609:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 17 previous similar messages [ 6004.197639] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6004.222269] LustreError: Skipped 2 previous similar messages [ 6006.933710] Lustre: server umount lustre-OST0001 complete [ 6013.924482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6013.929077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.935862] Lustre: Skipped 3 previous similar messages [ 6016.796761] Lustre: server umount lustre-MDT0000 complete [ 6020.618432] LustreError: 202647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189171 with bad export cookie 10210355351751179983 [ 6020.623732] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6020.629800] LustreError: 202647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6021.216252] Lustre: server umount lustre-MDT0001 complete [ 6045.882928] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:39:56 (1786189196) [ 6063.518721] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 6072.593240] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 6108.565647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.776355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6108.800148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6108.859912] Lustre: lustre-MDT0000: new disk, initializing [ 6108.916877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.929644] Lustre: Skipped 3 previous similar messages [ 6108.956954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6113.203588] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6124.530757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.652400] Lustre: 207855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6124.769351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6124.774180] Lustre: Skipped 1 previous similar message [ 6124.839874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6124.846757] Lustre: Skipped 9 previous similar messages [ 6124.889513] Lustre: lustre-MDT0001: new disk, initializing [ 6125.045374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6125.060224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6130.872692] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6140.176688] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6145.433612] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6151.803661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6152.315904] Lustre: lustre-OST0000: new disk, initializing [ 6152.323078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6152.341497] Lustre: 209827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6159.069533] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6159.929238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6159.941036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6160.094124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6168.521812] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6173.282364] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6173.500520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6178.275855] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6178.679927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6180.873565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.877469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.885536] Lustre: Skipped 2 previous similar messages [ 6186.799403] Lustre: server umount lustre-OST0000 complete [ 6197.510173] Lustre: server umount lustre-MDT0000 complete [ 6201.236993] LustreError: 207848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189351 with bad export cookie 10210355351751181544 [ 6201.243464] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6201.246513] LustreError: 207848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6201.836369] Lustre: server umount lustre-MDT0001 complete [ 6211.119816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.484384] LustreError: 212045:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.527651] LustreError: 212045:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 11 previous similar messages [ 6216.779614] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6225.104951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.593539] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6238.403612] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6244.485929] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6251.857530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.071209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.088147] Lustre: Skipped 4 previous similar messages [ 6259.600847] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6268.154209] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6273.276074] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6273.571971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6279.194223] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6279.528843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6283.745378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6283.749756] LustreError: Skipped 2 previous similar messages [ 6287.872925] Lustre: server umount lustre-OST0000 complete [ 6296.499445] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6301.108891] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6332.012390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.430177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.458421] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6339.060643] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6349.066497] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6351.360515] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6352.055486] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.075287] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6356.969350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6356.986901] LustreError: Skipped 1 previous similar message [ 6356.994805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.012783] Lustre: Skipped 4 previous similar messages [ 6360.754567] Lustre: server umount lustre-OST0000 complete [ 6371.064944] Lustre: server umount lustre-MDT0000 complete [ 6372.940308] LustreError: 212772:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.967617] LustreError: 212772:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 6374.310333] LustreError: 212026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189524 with bad export cookie 10210355351751182426 [ 6374.333546] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6374.635752] Lustre: server umount lustre-MDT0001 complete [ 6383.248525] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:45:33 (1786189533) [ 6393.173695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.408308] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6408.154186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.994199] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6420.128843] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6426.519332] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6434.671212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6441.025286] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6447.781301] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.715361] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6453.889835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6459.743112] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6459.896843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6468.120465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6468.263888] Lustre: lustre-OST0001: new disk, initializing [ 6468.266901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6468.271989] Lustre: 221384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6474.015013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6474.053655] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6474.165912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6475.144884] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6485.457374] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6490.502483] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6490.800701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6496.074519] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6496.230262] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6499.818520] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.832974] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6499.837366] LustreError: Skipped 2 previous similar messages [ 6499.858278] Lustre: Skipped 16 previous similar messages [ 6504.381427] Lustre: server umount lustre-OST0001 complete [ 6510.662831] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6510.828526] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6514.925385] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6515.332262] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6545.262517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6545.451582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6545.455979] Lustre: Skipped 5 previous similar messages [ 6550.529753] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6550.555555] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6550.561733] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6550.593288] Lustre: Skipped 1 previous similar message [ 6552.478098] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6561.653571] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6563.037884] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6570.931294] Lustre: server umount lustre-OST0001 complete [ 6592.904540] Lustre: server umount lustre-OST0000 complete [ 6602.926242] Lustre: server umount lustre-MDT0000 complete [ 6606.930825] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189757 with bad export cookie 10210355351751183392 [ 6606.945357] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.953051] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6613.535124] Lustre: server umount lustre-MDT0001 complete [ 6639.640767] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:49:49 (1786189789) [ 6646.924753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.208512] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6647.298664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6652.386143] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6661.316593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.372884] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6661.391153] Lustre: 226626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6666.361862] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6674.706561] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6680.437883] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6687.232985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.377095] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6693.700420] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6701.311550] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6711.844266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6711.937514] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6719.391771] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6726.466124] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6728.576175] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6732.927050] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6733.384492] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6738.337190] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6738.648152] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6744.782885] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c5bc63c3800.ost_server_uuid 50 [ 6746.441454] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c5bc63c3800.ost_server_uuid in IDLE state after 0 sec [ 6747.718691] Lustre: Permanently deactivating lustre-OST0001 [ 6747.755897] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6750.954603] Lustre: Permanently reactivating lustre-OST0001 [ 6757.857292] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6757.864263] LustreError: Skipped 4 previous similar messages [ 6757.866829] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.874401] Lustre: Skipped 11 previous similar messages [ 6761.753326] Lustre: server umount lustre-OST0001 complete [ 6763.516550] LustreError: 228471:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6763.552034] LustreError: 228471:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 38 previous similar messages [ 6772.382889] Lustre: server umount lustre-OST0000 complete [ 6782.339420] Lustre: server umount lustre-MDT0000 complete [ 6786.580869] LustreError: 226621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189937 with bad export cookie 10210355351751184680 [ 6786.587299] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6786.606213] LustreError: 226621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6786.971141] Lustre: server umount lustre-MDT0001 complete [ 6817.588609] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:52:47 (1786189967) [ 6826.101879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.289882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6826.316574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6826.328383] Lustre: Skipped 5 previous similar messages [ 6830.748665] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6839.863431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.920710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6839.938204] Lustre: 233321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6845.850839] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6853.515792] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6860.532801] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6867.030380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.175568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6867.194169] Lustre: Permanently deactivating lustre-OST0000 [ 6867.204465] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6873.140770] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6873.271604] LustreError: 235534:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6873.278667] LustreError: 235534:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6873.363161] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6882.209144] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6891.166539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.213454] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6896.768537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6898.868862] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 6906.468609] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6909.130299] Lustre: Permanently reactivating lustre-OST0000 [ 6909.204435] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6909.213450] Lustre: Skipped 2 previous similar messages [ 6915.102175] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6915.102283] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000afa7eeb9, cur 1786190065 deadline 1786190054 last 1786190024 [ 6915.118739] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6915.138405] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6915.162309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6927.208765] Lustre: server umount lustre-OST0001 complete [ 6937.316899] Lustre: server umount lustre-OST0000 complete [ 6948.634393] Lustre: server umount lustre-MDT0000 complete [ 6952.027093] LustreError: 233315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190102 with bad export cookie 10210355351751186325 [ 6952.033958] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6952.051080] LustreError: 233315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6952.356140] Lustre: server umount lustre-MDT0001 complete [ 6965.203563] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 6967.528753] Key type lgssc unregistered [ 6967.740632] LNet: 238208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6967.752872] LNetError: 238208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6968.805805] LNet: Removed LNI 192.168.203.110@tcp [ 6969.454956] Key type .llcrypt unregistered [ 6969.457707] Key type ._llcrypt unregistered [ 6997.985773] Key type ._llcrypt registered [ 6997.987889] Key type .llcrypt registered [ 6998.085478] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 6999.872509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7001.083071] alg: No test for adler32 (adler32-zlib) [ 7002.743103] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7003.065534] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 7004.784171] Key type lgssc registered [ 7006.354756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7015.848787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7015.862574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.385982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.441120] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7026.730172] LustreError: 239675:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.742368] LustreError: 239675:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7027.763133] LustreError: 239676:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.191871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.337835] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7031.362123] Lustre: 239663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7031.374540] Lustre: Permanently deactivating lustre-MDT0001 [ 7031.383878] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7031.608513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.134265] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7035.151420] LustreError: 239706:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7035.170690] LustreError: 239706:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7035.184672] LustreError: 239706:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7036.185920] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7044.530106] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7050.527351] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7057.753449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.122544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.255739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7063.280133] LustreError: 241516:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.404867] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7068.652700] LustreError: 241516:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.004546] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7073.766510] LustreError: 241518:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7076.861510] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7079.562160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.702835] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.107431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7085.145106] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7087.223506] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7096.193764] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7099.078684] Lustre: Permanently reactivating lustre-MDT0001 [ 7099.096325] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7099.104936] Lustre: Skipped 2 previous similar messages [ 7104.202862] Lustre: lustre-MDT0001: Client 818f9b40-6fb1-4d2f-9e2b-a62a00cb6189 (at 192.168.203.10@tcp) reconnecting [ 7111.016250] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7112.173572] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7118.124421] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7118.669978] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7123.373928] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7123.575824] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7125.805444] Lustre: Permanently deactivating lustre-MDT0001 [ 7125.839373] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7125.856398] Lustre: Skipped 2 previous similar messages [ 7134.227547] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7144.420509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7144.425339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.439079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.467017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.472188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.480012] Lustre: Skipped 2 previous similar messages [ 7151.894697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.900376] Lustre: Skipped 6 previous similar messages [ 7157.216499] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7157.219678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7157.242669] Lustre: Skipped 3 previous similar messages [ 7157.632877] Lustre: server umount lustre-MDT0000 complete [ 7161.784021] LustreError: 240427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190312 with bad export cookie 16323947955344796794 [ 7161.786851] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7161.794547] LustreError: 240427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7162.070594] Lustre: server umount lustre-MDT0001 complete [ 7168.711799] Lustre: server umount lustre-OST0000 complete [ 7180.885922] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:58:50 (1786190330) [ 7191.257795] Lustre: server umount lustre-OST0001 complete [ 7197.598315] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 7207.648498] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 7240.695522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.911421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7240.922107] Lustre: Skipped 2 previous similar messages [ 7240.945955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7241.036511] Lustre: lustre-MDT0000: new disk, initializing [ 7241.120254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.140482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7245.348298] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7256.279051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.377202] Lustre: 247985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7256.418655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7256.422225] Lustre: Skipped 1 previous similar message [ 7256.470923] Lustre: lustre-MDT0001: new disk, initializing [ 7256.562235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.610980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7256.626537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7261.496517] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7270.867647] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7277.061432] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.318052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.534135] Lustre: lustre-OST0000: new disk, initializing [ 7284.537552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7284.543169] Lustre: 249952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7284.628533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.224809] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7294.704039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7294.710770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7294.924592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7302.337345] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7304.411398] LustreError: 247993:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7312.861719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7313.127185] Lustre: lustre-OST0001: new disk, initializing [ 7313.132396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7313.144540] Lustre: 251081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7314.512091] LustreError: 247993:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7314.548150] LustreError: 249331:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7319.237040] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7319.582532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7319.621412] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7319.908413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7324.640274] LustreError: 249331:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7329.094272] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7330.842766] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7335.100480] Lustre: lustre-OST0001: Not available for connect from 192.168.203.10@tcp (stopping) [ 7335.394511] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7335.405315] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.416076] LustreError: Skipped 1 previous similar message [ 7335.436186] Lustre: Skipped 1 previous similar message [ 7338.630943] Lustre: server umount lustre-OST0001 complete [ 7340.216117] LustreError: 249937:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.249834] LustreError: 249937:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7355.877669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.879812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7355.884348] Lustre: Skipped 1 previous similar message [ 7355.949036] Lustre: Skipped 3 previous similar messages [ 7361.014883] LustreError: 249937:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7361.038626] LustreError: 249937:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 9 previous similar messages [ 7361.205820] Lustre: server umount lustre-OST0000 complete [ 7366.125780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.133580] Lustre: Skipped 1 previous similar message [ 7370.946104] Lustre: server umount lustre-MDT0000 complete [ 7374.648651] LustreError: 247977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190525 with bad export cookie 16323947955344799832 [ 7374.650448] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7374.667801] LustreError: 247977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7381.172358] Lustre: server umount lustre-MDT0001 complete [ 7391.307912] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 7393.458559] Key type lgssc unregistered [ 7393.657963] LNet: 253113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7393.665056] LNetError: 253113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7393.706656] LNet: Removed LNI 192.168.203.110@tcp [ 7394.391782] Key type .llcrypt unregistered [ 7394.394177] Key type ._llcrypt unregistered [ 7439.880479] Key type ._llcrypt registered [ 7439.882405] Key type .llcrypt registered [ 7440.094267] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 7442.987317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7443.056054] alg: No test for adler32 (adler32-zlib) [ 7444.484581] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7445.092698] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 7446.832579] Key type lgssc registered [ 7448.297465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7456.464681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7456.480213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.900520] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7457.951746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7458.953276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7463.497810] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7472.563141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.635169] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7472.667240] Lustre: 255323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7473.042106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7477.082348] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7484.265202] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7490.690320] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7497.528569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.662570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7498.167742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.067833] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7505.209377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7513.353345] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7566.819573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7566.826985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.830093] LustreError: Skipped 1 previous similar message [ 7566.839574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.844862] Lustre: Skipped 1 previous similar message [ 7566.891748] Lustre: Skipped 1 previous similar message [ 7568.741276] Lustre: server umount lustre-OST0000 complete [ 7572.731266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7584.576328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.987177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.058110] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7587.098702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7587.106901] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7587.161931] Lustre: Skipped 1 previous similar message [ 7592.511435] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7602.104478] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7643.618358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7643.618508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7643.619441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7643.626989] Lustre: Skipped 1 previous similar message [ 7643.649086] LustreError: Skipped 1 previous similar message [ 7646.532750] Lustre: server umount lustre-OST0000 complete [ 7652.323845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7652.332886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.357826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.362552] Lustre: Skipped 1 previous similar message [ 7656.403692] Lustre: server umount lustre-MDT0000 complete [ 7660.381878] LustreError: 255314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190811 with bad export cookie 12263577501238494752 [ 7660.384155] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7660.389189] LustreError: 255314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7660.630805] Lustre: server umount lustre-MDT0001 complete [ 7671.218533] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 7673.402995] Key type lgssc unregistered [ 7673.650393] LNet: 260751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7673.657263] LNetError: 260751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7673.674062] LNet: Removed LNI 192.168.203.110@tcp [ 7674.365350] Key type .llcrypt unregistered [ 7674.369239] Key type ._llcrypt unregistered [ 7699.127793] Key type ._llcrypt registered [ 7699.129801] Key type .llcrypt registered [ 7699.238977] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 7700.619320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7700.689438] alg: No test for adler32 (adler32-zlib) [ 7702.208292] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7702.425129] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 7704.120533] Key type lgssc registered [ 7705.448738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7712.606092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7712.664656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.102461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.508399] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7723.523407] LustreError: 262168:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7723.559365] LustreError: 262168:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7728.617323] LustreError: 262169:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.514899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.871083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7735.122659] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7742.244916] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7747.894596] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7754.475968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.702321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.850154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7759.906426] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7767.130086] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7791.588467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7791.600066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.624908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.659843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7794.660340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.667410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.703822] Lustre: Skipped 1 previous similar message [ 7796.184657] Lustre: server umount lustre-OST0000 complete [ 7799.778021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.781016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.785568] Lustre: Skipped 1 previous similar message [ 7799.791832] Lustre: Skipped 1 previous similar message [ 7804.905209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7804.910490] Lustre: Skipped 1 previous similar message [ 7806.015686] Lustre: server umount lustre-MDT0000 complete [ 7809.878334] LustreError: 262147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190960 with bad export cookie 9602523922343367051 [ 7809.879434] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7809.893406] LustreError: 262147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7810.189555] Lustre: server umount lustre-MDT0001 complete [ 7820.393960] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 7822.584524] Key type lgssc unregistered [ 7822.833789] LNet: 266475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7822.837651] LNetError: 266475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7822.854168] LNet: Removed LNI 192.168.203.110@tcp [ 7823.693880] Key type .llcrypt unregistered [ 7823.697227] Key type ._llcrypt unregistered [ 7843.500077] Key type ._llcrypt registered [ 7843.502700] Key type .llcrypt registered [ 7843.709897] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 7846.263544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7846.337062] alg: No test for adler32 (adler32-zlib) [ 7847.361966] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7847.556868] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 7849.208163] Key type lgssc registered [ 7850.244482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7858.189391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7858.201667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.567485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.170356] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7869.923350] LustreError: 267699:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.940331] LustreError: 267699:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7871.116883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.629758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7875.613350] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7881.335479] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7886.263389] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7892.880685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.303562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.011219] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7904.641249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7906.357423] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7920.097867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.099665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7920.105383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7920.111063] Lustre: Skipped 1 previous similar message [ 7920.679966] Lustre: server umount lustre-OST0000 complete [ 7925.217927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7925.220922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7925.232682] Lustre: Skipped 1 previous similar message [ 7925.241534] Lustre: Skipped 2 previous similar messages [ 7930.339362] LustreError: 267699:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7930.388676] Lustre: server umount lustre-MDT0000 complete [ 7933.742307] LustreError: 267681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191084 with bad export cookie 9310641471696056065 [ 7933.744452] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7933.748619] LustreError: 267681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7934.197522] Lustre: server umount lustre-MDT0001 complete [ 7945.114359] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 7947.406990] Key type lgssc unregistered [ 7947.667590] LNet: 271381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7947.673399] LNetError: 271381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7948.727879] LNet: Removed LNI 192.168.203.110@tcp [ 7949.424381] Key type .llcrypt unregistered [ 7949.426549] Key type ._llcrypt unregistered [ 7973.771964] Key type ._llcrypt registered [ 7973.773552] Key type .llcrypt registered [ 7973.876593] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 7974.841284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.915493] alg: No test for adler32 (adler32-zlib) [ 7976.065593] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7976.312886] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 7977.944743] Key type lgssc registered [ 7978.837660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.032939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7986.062686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.358957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.825322] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 7997.925551] LustreError: 272796:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.943914] LustreError: 272796:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 8000.000865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.418496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.902339] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8012.045305] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8017.894910] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8023.700371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.907753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8029.259706] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8033.279082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8037.033242] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8058.853628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8058.874921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8058.877515] Lustre: Skipped 1 previous similar message [ 8058.878106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8063.972055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8063.975715] Lustre: Skipped 1 previous similar message [ 8065.216274] Lustre: server umount lustre-OST0000 complete [ 8069.090285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.092551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8069.101718] Lustre: Skipped 1 previous similar message [ 8069.110570] Lustre: Skipped 1 previous similar message [ 8074.209485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8074.214065] Lustre: Skipped 1 previous similar message [ 8074.909485] Lustre: server umount lustre-MDT0000 complete [ 8078.527728] LustreError: 272778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191229 with bad export cookie 746091651244450490 [ 8078.538742] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8078.548649] LustreError: 272778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8078.745933] Lustre: server umount lustre-MDT0001 complete [ 8089.613385] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 8091.855753] Key type lgssc unregistered [ 8092.062621] LNet: 277106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8092.074832] LNetError: 277106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8092.105637] LNet: Removed LNI 192.168.203.110@tcp [ 8092.828780] Key type .llcrypt unregistered [ 8092.830845] Key type ._llcrypt unregistered [ 8110.696331] Key type ._llcrypt registered [ 8110.698811] Key type .llcrypt registered [ 8110.852661] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8111.818411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8112.143298] alg: No test for adler32 (adler32-zlib) [ 8113.278837] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 8113.565621] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 8115.216924] Key type lgssc registered [ 8116.027402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8123.434665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8123.457682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.782332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.465584] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8135.139317] LustreError: 278342:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.156796] LustreError: 278342:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 8135.699737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.005774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8139.662424] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8146.781137] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8152.207129] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8158.166798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.382582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8163.892543] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8170.461595] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8171.699518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8181.217770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8181.222453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.233659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.245657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8182.247636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.249819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.277288] Lustre: Skipped 1 previous similar message [ 8186.040317] Lustre: server umount lustre-OST0000 complete [ 8192.482599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8192.483046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.492729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.540909] Lustre: Skipped 1 previous similar message [ 8195.595739] Lustre: server umount lustre-MDT0000 complete [ 8197.608144] LustreError: 278347:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.625974] LustreError: 278347:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 8198.729149] LustreError: 278308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191349 with bad export cookie 6104264300429859306 [ 8198.731541] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8198.735673] LustreError: 278308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8199.022037] Lustre: server umount lustre-MDT0001 complete [ 8207.244887] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 8209.648880] Key type lgssc unregistered [ 8209.871898] LNet: 282046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8209.885521] LNetError: 282046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8209.901951] LNet: Removed LNI 192.168.203.110@tcp [ 8210.765515] Key type .llcrypt unregistered [ 8210.767804] Key type ._llcrypt unregistered [ 8221.315434] Key type ._llcrypt registered [ 8221.317541] Key type .llcrypt registered [ 8221.387494] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8235.039500] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 8247.268310] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8248.737509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8248.750475] alg: No test for adler32 (adler32-zlib) [ 8249.738560] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 8249.937469] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 8251.592183] Key type lgssc registered [ 8252.466634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8276.178968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8276.208336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8277.390475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8277.445515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8277.499815] Lustre: lustre-MDT0000: new disk, initializing [ 8277.569125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.594965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8280.662693] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8288.617843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.710605] Lustre: 285580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8288.782376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8288.786516] Lustre: Skipped 1 previous similar message [ 8288.839173] Lustre: lustre-MDT0001: new disk, initializing [ 8288.906024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8288.934399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8288.946628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8292.434163] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8299.372559] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8303.190585] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.043991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.245533] Lustre: lustre-OST0000: new disk, initializing [ 8308.250296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8308.258195] Lustre: 287545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8308.328049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.517627] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8318.058750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8318.074211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8318.173733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8319.453037] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8323.005345] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8323.170352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8326.725747] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8326.924767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8328.675268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.678702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8328.688967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.694651] Lustre: Skipped 1 previous similar message [ 8333.794177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8333.797509] Lustre: Skipped 1 previous similar message [ 8334.482581] Lustre: server umount lustre-OST0000 complete [ 8338.913858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8338.917136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8338.930570] Lustre: Skipped 2 previous similar messages [ 8343.176550] Lustre: server umount lustre-MDT0000 complete [ 8345.892323] LustreError: 285574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191496 with bad export cookie 17839518217826764981 [ 8345.901466] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8345.908963] LustreError: 285574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8346.138668] Lustre: server umount lustre-MDT0001 complete [ 8352.493320] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:18:22 (1786191502) [ 8357.871747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.148786] LustreError: 289955:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.166361] LustreError: 289955:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 8358.206940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.384776] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8363.494662] LustreError: 289956:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.588613] LustreError: 289955:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.617975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.841653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8371.397526] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8376.745745] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8380.854854] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8385.361774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.609889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.353447] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8395.879616] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8413.666630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8413.673906] LustreError: Skipped 1 previous similar message [ 8413.675585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.681910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8413.686034] Lustre: Skipped 1 previous similar message [ 8413.693737] Lustre: Skipped 2 previous similar messages [ 8417.645379] Lustre: server umount lustre-OST0000 complete [ 8420.835080] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8420.838062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.844571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8426.329478] Lustre: server umount lustre-MDT0000 complete [ 8429.028120] LustreError: 289956:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.038494] LustreError: 289956:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 8429.149831] LustreError: 290692:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8429.151273] LustreError: 289936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191579 with bad export cookie 17839518217826765863 [ 8429.152129] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8429.182208] LustreError: 289936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8429.405571] Lustre: server umount lustre-MDT0001 complete [ 8437.764946] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 8439.702601] Key type lgssc unregistered [ 8439.889650] LNet: 293512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8439.897936] LNetError: 293512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8439.908492] LNet: Removed LNI 192.168.203.110@tcp [ 8440.504703] Key type .llcrypt unregistered [ 8440.506952] Key type ._llcrypt unregistered [ 8458.111864] Key type ._llcrypt registered [ 8458.115351] Key type .llcrypt registered [ 8458.193516] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 8467.726300] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8468.390298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8468.407232] alg: No test for adler32 (adler32-zlib) [ 8469.345951] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 8469.499612] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 8471.120156] Key type lgssc registered [ 8471.948207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8502.526933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8502.545183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.704992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8503.725810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8503.779133] Lustre: lustre-MDT0000: new disk, initializing [ 8503.834632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8503.853933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8506.766445] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8513.281474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.368571] Lustre: 297314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8513.414664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8513.419754] Lustre: Skipped 1 previous similar message [ 8513.499063] Lustre: lustre-MDT0001: new disk, initializing [ 8513.582716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8513.605201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8513.618382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8516.470120] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8522.720609] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8527.633441] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8531.962089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8532.158346] Lustre: lustre-OST2710: new disk, initializing [ 8532.161418] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8532.165936] Lustre: 299283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8532.230272] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8535.901634] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8539.165877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8539.175128] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8539.234678] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8541.842543] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8545.574918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8545.717417] Lustre: lustre-OST03e8: new disk, initializing [ 8545.720121] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8545.724783] Lustre: 300348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8545.785467] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8549.545392] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8553.909808] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.539563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8554.546168] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8554.617356] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8558.776153] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8558.890277] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8561.508518] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8561.632814] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8564.083721] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8564.205790] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8566.676964] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8566.828793] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8569.828231] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.830786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.831361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8569.834300] Lustre: Skipped 3 previous similar messages [ 8569.846550] Lustre: Skipped 3 previous similar messages [ 8574.947165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.950023] Lustre: Skipped 3 previous similar messages [ 8575.081563] Lustre: server umount lustre-MDT0000 complete [ 8577.311532] LustreError: 297308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191727 with bad export cookie 13981521850568266187 [ 8577.314383] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.323494] LustreError: 297308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8577.515679] Lustre: server umount lustre-MDT0001 complete [ 8590.308767] Lustre: server umount lustre-OST2710 complete [ 8603.127156] Lustre: server umount lustre-OST03e8 complete [ 8614.602536] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 8621.268422] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8644.056510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.254644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8644.275581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8644.342266] Lustre: lustre-MDT0000: new disk, initializing [ 8644.400529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8644.411660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8647.464136] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8654.695110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.769361] Lustre: 305587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8654.814300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8654.818610] Lustre: Skipped 1 previous similar message [ 8654.868779] Lustre: lustre-MDT0001: new disk, initializing [ 8654.922640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8654.949321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8654.967459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8657.692212] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8663.934388] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8667.378647] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8671.877171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.044505] Lustre: lustre-OST0000: new disk, initializing [ 8672.047514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8672.055303] Lustre: 307555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8672.133228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.570656] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8678.431403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8678.443422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8678.520505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8681.038424] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8683.538577] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8683.672620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8686.234716] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8686.349866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8688.610629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.611989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.624576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.627430] Lustre: Skipped 1 previous similar message [ 8693.487107] Lustre: server umount lustre-OST0000 complete [ 8695.778483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8695.783325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.790090] Lustre: Skipped 1 previous similar message [ 8695.793253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.852987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.889978] Lustre: server umount lustre-MDT0000 complete [ 8703.950848] LustreError: 307554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191854 with bad export cookie 13981521850568267867 [ 8703.956454] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8703.964136] LustreError: 307554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8704.150278] Lustre: server umount lustre-MDT0001 complete [ 8709.055084] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:24:19 (1786191859) [ 8709.984266] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8710.994514] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:24:21 (1786191861) [ 8712.884770] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8728.865945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.004928] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8729.015594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8729.096086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.991450] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8737.091212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.165855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8737.185188] Lustre: 311406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8740.072526] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8758.503679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.616181] Lustre: Denying initial registration attempt from nid 192.168.203.110@tcp, specified as failover [ 8758.621250] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8758.626364] LustreError: 312857:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8758.629760] LustreError: 312857:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8758.633133] LustreError: 312857:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8758.673579] Lustre: server umount lustre-OST0000 complete [ 8758.675895] LustreError: 312857:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8762.849286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8762.856107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.867593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.874103] Lustre: Skipped 2 previous similar messages [ 8768.506564] Lustre: server umount lustre-MDT0000 complete [ 8770.542062] LustreError: 311400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191921 with bad export cookie 13981521850568268749 [ 8770.544181] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8770.550855] LustreError: 311400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8770.747115] Lustre: server umount lustre-MDT0001 complete [ 8776.517474] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 8777.965556] Key type lgssc unregistered [ 8778.129774] LNet: 313982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8778.134574] LNetError: 313982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8778.147676] LNet: Removed LNI 192.168.203.110@tcp [ 8778.596112] Key type .llcrypt unregistered [ 8778.599532] Key type ._llcrypt unregistered [ 8784.377879] Key type ._llcrypt registered [ 8784.379871] Key type .llcrypt registered [ 8784.449826] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8785.006629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8785.034781] alg: No test for adler32 (adler32-zlib) [ 8785.941607] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 8786.104808] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 8787.728829] Key type lgssc registered [ 8788.506683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8819.892493] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8824.118619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8824.135046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.289424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8825.308120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8825.400852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8827.724474] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8832.372646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.425848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8832.442116] Lustre: 316722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8832.602350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8835.255900] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8839.507926] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8842.826598] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8860.004263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8860.114544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8860.212802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8863.086640] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8866.431522] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8872.419719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8872.419759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.421397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8872.425595] LustreError: Skipped 1 previous similar message [ 8873.591300] Lustre: server umount lustre-OST0000 complete [ 8877.542109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.544509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8877.550307] Lustre: Skipped 2 previous similar messages [ 8877.555382] Lustre: Skipped 1 previous similar message [ 8881.797121] Lustre: server umount lustre-MDT0000 complete [ 8883.694989] LustreError: 316716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192034 with bad export cookie 11990424800337676765 [ 8883.698067] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.707247] LustreError: 316716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8883.867945] Lustre: server umount lustre-MDT0001 complete [ 8889.522678] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 8891.038177] Key type lgssc unregistered [ 8891.215564] LNet: 320295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8891.222535] LNetError: 320295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8891.235631] LNet: Removed LNI 192.168.203.110@tcp [ 8891.706172] Key type .llcrypt unregistered [ 8891.709328] Key type ._llcrypt unregistered [ 8904.990363] Key type ._llcrypt registered [ 8904.992108] Key type .llcrypt registered [ 8905.052493] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 8905.484989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8905.522251] alg: No test for adler32 (adler32-zlib) [ 8906.461505] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 8906.587723] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 8908.200463] Key type lgssc registered [ 8908.781991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8912.483482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8912.494489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.685550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.591206] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8919.470309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.670476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8921.667671] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8925.229312] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8927.924682] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8933.027889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8933.178221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.620374] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8938.837409] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8948.313321] LustreError: 324268:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (00000000a906b37b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8949.732444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.733191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.743242] Lustre: Skipped 2 previous similar messages [ 8949.746633] Lustre: Skipped 1 previous similar message [ 8950.437765] Lustre: server umount lustre-MDT0000 complete [ 8950.453189] LustreError: 321686:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.460936] LustreError: 321686:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 8952.073616] LustreError: 323533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192102 with bad export cookie 12395696301519073963 [ 8952.075738] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8952.080968] LustreError: 323533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8954.849147] LustreError: 321687:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8954.850851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.868147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8954.871618] Lustre: Skipped 1 previous similar message [ 8957.857223] LustreError: 322885:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.857265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8957.868497] LustreError: 322885:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 8957.876254] Lustre: Skipped 1 previous similar message [ 8958.293591] Lustre: server umount lustre-MDT0001 complete [ 8961.160278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8965.477663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.352816] LustreError: 320999:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ecfa0e4f x1872958122384768/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8976.518682] LustreError: 323521:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.564297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8976.599942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8978.538306] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8981.667591] LustreError: 323520:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8981.679797] LustreError: 323520:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 8982.609116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.804243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8984.674225] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 8986.917365] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8987.772562] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8992.612302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 8995.175486] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8997.966492] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 9002.468966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.470028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9002.478090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9002.485603] Lustre: Skipped 1 previous similar message [ 9007.586760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9014.240131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9014.320260] Lustre: server umount lustre-OST0000 complete [ 9017.828264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.831160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.837463] Lustre: Skipped 2 previous similar messages [ 9017.841238] Lustre: Skipped 3 previous similar messages [ 9022.594219] Lustre: server umount lustre-MDT0000 complete [ 9022.946061] LustreError: 325277:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.954105] LustreError: 325277:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9024.424698] LustreError: 323533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192175 with bad export cookie 12395696301519081117 [ 9024.425361] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.435367] LustreError: 323533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9024.619754] Lustre: server umount lustre-MDT0001 complete [ 9030.015378] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 9031.496904] Key type lgssc unregistered [ 9031.671471] LNet: 327952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9031.675656] LNetError: 327952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9031.683792] LNet: Removed LNI 192.168.203.110@tcp [ 9032.168232] Key type .llcrypt unregistered [ 9032.169774] Key type ._llcrypt unregistered [ 9045.998864] Key type ._llcrypt registered [ 9046.001496] Key type .llcrypt registered [ 9046.091688] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 9046.593186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9046.642689] alg: No test for adler32 (adler32-zlib) [ 9047.607135] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9047.767089] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 9049.376209] Key type lgssc registered [ 9050.072166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9054.674980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9054.691031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.940715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.053809] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9062.555899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.777441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9064.997121] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9069.019992] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9072.121385] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9078.242631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.243555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9078.244219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9078.252057] Lustre: Skipped 1 previous similar message [ 9080.045742] Lustre: server umount lustre-MDT0000 complete [ 9081.995528] LustreError: 329327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192232 with bad export cookie 3969041768792329707 [ 9081.999616] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9082.003872] LustreError: 329327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9082.165192] Lustre: server umount lustre-MDT0001 complete [ 9090.415876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.590778] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9090.613658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9090.714213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.019781] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9097.198173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.263707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9097.286659] Lustre: 331930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9097.435960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9099.569976] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9103.568393] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9106.717269] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9113.382374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9113.483774] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9113.489442] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9113.493916] LustreError: 333811:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9113.497292] LustreError: 333811:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9113.501261] LustreError: 333811:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9113.542829] Lustre: server umount lustre-OST0000 complete [ 9113.545489] LustreError: 333811:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9120.678771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.770534] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9120.873879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.361914] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9128.631495] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9129.033577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9133.180820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.249467] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9133.307864] Lustre: lustre-OST0001: new disk, initializing [ 9133.311106] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9133.315748] Lustre: 335358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9133.393963] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9134.869884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9134.876798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9134.969272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9137.098140] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9141.347652] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9145.319195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9145.320605] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9145.323305] LustreError: Skipped 1 previous similar message [ 9145.333826] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9145.340349] Lustre: Skipped 2 previous similar messages [ 9149.011388] Lustre: server umount lustre-OST0001 complete [ 9150.441198] LustreError: 334970:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9150.463843] LustreError: 334970:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9154.529210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9154.529553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.534398] LustreError: Skipped 1 previous similar message [ 9154.535687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9154.540593] Lustre: Skipped 2 previous similar messages [ 9154.552584] Lustre: Skipped 1 previous similar message [ 9155.562126] LustreError: 334970:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9155.577201] LustreError: 334970:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9157.247485] Lustre: server umount lustre-OST0000 complete [ 9160.674836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.676322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.680831] Lustre: Skipped 1 previous similar message [ 9160.683910] Lustre: Skipped 2 previous similar messages [ 9165.462955] Lustre: server umount lustre-MDT0000 complete [ 9165.793658] LustreError: 331943:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9165.804558] LustreError: 331943:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9167.459774] LustreError: 331924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192318 with bad export cookie 3969041768792330239 [ 9167.463309] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9167.465439] LustreError: 331924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9167.680440] Lustre: server umount lustre-MDT0001 complete [ 9173.222611] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 9174.638181] Key type lgssc unregistered [ 9174.801274] LNet: 337315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9174.809788] LNetError: 337315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9174.825466] LNet: Removed LNI 192.168.203.110@tcp [ 9175.257454] Key type .llcrypt unregistered [ 9175.259521] Key type ._llcrypt unregistered [ 9219.074226] Key type ._llcrypt registered [ 9219.076176] Key type .llcrypt registered [ 9219.144128] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 9227.718577] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 9228.240466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.256166] alg: No test for adler32 (adler32-zlib) [ 9229.170395] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9229.319539] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 9230.920204] Key type lgssc registered [ 9231.588715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9249.444294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9249.463761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.594489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9250.613273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9250.660971] Lustre: lustre-MDT0000: new disk, initializing [ 9250.707807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9250.718276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9252.921710] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9259.355751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.414809] Lustre: 342222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9259.459670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9259.467941] Lustre: Skipped 1 previous similar message [ 9259.510769] Lustre: lustre-MDT0001: new disk, initializing [ 9259.572465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9259.598616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9259.608647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9262.183153] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9267.750615] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9271.045359] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9274.718478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9274.869421] Lustre: lustre-OST0000: new disk, initializing [ 9274.872799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9274.876115] Lustre: 344189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9274.920768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9278.221839] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9284.187484] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9284.672100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9284.678517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9284.767085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9286.773810] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9286.904733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9289.232445] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9289.350569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9290.525522] Lustre: server umount lustre-OST0000 complete [ 9295.330261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.330260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.330758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9295.331246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9300.451799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9300.454801] Lustre: Skipped 2 previous similar messages [ 9302.541361] Lustre: server umount lustre-MDT0000 complete [ 9304.621128] LustreError: 342216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192455 with bad export cookie 4025044295442221402 [ 9304.621705] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9304.633967] LustreError: 342216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9314.880427] Lustre: server umount lustre-MDT0001 complete [ 9319.652185] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:34:30 (1786192470) [ 9323.686575] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:34:34 (1786192474) [ 9328.213962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.530619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.660703] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9333.730279] LustreError: 346944:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.738916] LustreError: 346944:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 9335.034477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.218230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9337.208462] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9340.650525] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9343.612162] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9347.136587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9347.342987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9350.535343] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9354.616196] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9364.479314] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9365.392440] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9366.194662] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9367.079076] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9368.020033] Lustre: Failing over lustre-MDT0000 [ 9368.158256] Lustre: server umount lustre-MDT0000 complete [ 9370.084264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9370.086535] LustreError: 349381:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.104259] Lustre: Skipped 2 previous similar messages [ 9372.345502] LustreError: 349381:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9372.354713] LustreError: 349381:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 9375.203193] LustreError: 346944:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.212879] LustreError: 346944:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 9380.322194] LustreError: 346943:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.331893] LustreError: 346943:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9382.687358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.757838] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9382.880238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.911160] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9385.063381] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9387.693683] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9388.012254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9388.028755] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9388.052460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9391.008313] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9391.820660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9393.979822] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9394.982282] Lustre: Failing over lustre-MDT0000 [ 9395.187767] Lustre: server umount lustre-MDT0000 complete [ 9396.198587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.200142] LustreError: 346939:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9396.208180] Lustre: Skipped 2 previous similar messages [ 9396.222259] LustreError: 346939:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9409.692408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9409.756486] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9409.929126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9411.778254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9412.089455] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9415.142914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9415.146323] Lustre: Skipped 2 previous similar messages [ 9415.148529] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9415.168118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9417.486289] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9418.189568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9425.379942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9425.381925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9425.384830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9425.397286] Lustre: Skipped 1 previous similar message [ 9426.932184] Lustre: server umount lustre-MDT0000 complete [ 9428.571571] LustreError: 346924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192579 with bad export cookie 4025044295442224069 [ 9428.577637] LustreError: 346924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9428.580378] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9428.750595] Lustre: server umount lustre-MDT0001 complete [ 9440.695408] Lustre: server umount lustre-OST0000 complete [ 9447.312069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.535055] LustreError: 353272:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.543373] LustreError: 353272:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 9 previous similar messages [ 9447.584822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.588134] Lustre: Skipped 1 previous similar message [ 9449.635956] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9453.697312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.766337] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9459.054346] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9461.395760] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.284580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9466.648244] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9469.454996] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9472.615335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9483.233116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9483.238951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9483.245689] Lustre: Skipped 1 previous similar message [ 9483.249535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9483.253029] Lustre: Skipped 2 previous similar messages [ 9487.845854] Lustre: server umount lustre-OST0000 complete [ 9489.889316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9489.893815] LustreError: Skipped 1 previous similar message [ 9489.897452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9489.899868] Lustre: Skipped 2 previous similar messages [ 9494.498850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.505543] Lustre: Skipped 2 previous similar messages [ 9495.799809] Lustre: server umount lustre-MDT0000 complete [ 9497.726709] LustreError: 353254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192648 with bad export cookie 4025044295442224811 [ 9497.730561] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9497.737436] LustreError: 353254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9497.882678] Lustre: server umount lustre-MDT0001 complete [ 9503.556830] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 9504.964859] Key type lgssc unregistered [ 9505.127510] LNet: 356826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9505.133048] LNetError: 356826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9505.143991] LNet: Removed LNI 192.168.203.110@tcp [ 9505.574489] Key type .llcrypt unregistered [ 9505.576596] Key type ._llcrypt unregistered [ 9519.518656] Key type ._llcrypt registered [ 9519.520732] Key type .llcrypt registered [ 9519.579342] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 9520.005978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9520.025472] alg: No test for adler32 (adler32-zlib) [ 9520.917578] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9521.027042] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 9522.616244] Key type lgssc registered [ 9523.190800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9527.018698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9527.030518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9528.254742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9530.602645] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9534.880764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9535.054538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9536.867534] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9540.333952] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9543.358391] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9546.898231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9547.085763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.356127] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9553.259759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9554.588939] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9564.643852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9564.645213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9564.654432] Lustre: Skipped 2 previous similar messages [ 9564.659556] Lustre: Skipped 1 previous similar message [ 9569.763083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9569.767012] Lustre: Skipped 3 previous similar messages [ 9570.804304] Lustre: server umount lustre-MDT0000 complete [ 9572.705441] LustreError: 358300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192723 with bad export cookie 4087527702867187217 [ 9572.708894] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.711690] LustreError: 358300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9572.889703] Lustre: server umount lustre-MDT0001 complete [ 9585.153991] Lustre: server umount lustre-OST0000 complete [ 9591.811676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9592.047053] LustreError: 362026:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.064665] LustreError: 362026:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 9592.096783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9593.982443] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9597.410330] LustreError: 362027:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9598.137363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.327429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9600.365207] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9603.709803] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9606.227600] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9609.490608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9609.673527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9612.319128] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9615.570579] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9616.874636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9629.153080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9629.157854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.165302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9633.250222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9633.251034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9633.260258] Lustre: Skipped 1 previous similar message [ 9634.721409] Lustre: server umount lustre-MDT0000 complete [ 9636.107310] LustreError: 364703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192786 with bad export cookie 4087527702867188316 [ 9636.107919] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9636.112685] LustreError: 364703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9636.253802] Lustre: server umount lustre-MDT0001 complete [ 9648.115930] Lustre: server umount lustre-OST0000 complete [ 9655.360575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.588838] LustreError: 365931:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9655.638360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.599487] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9660.900498] LustreError: 365932:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.671421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.853240] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9667.380063] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9670.011624] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9673.435799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9676.516993] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9680.160344] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9680.808420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9691.105312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9691.108709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9691.116437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9691.120282] Lustre: Skipped 2 previous similar messages [ 9692.132240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9693.296767] Lustre: server umount lustre-OST0000 complete [ 9697.251663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.260891] Lustre: Skipped 2 previous similar messages [ 9701.164582] Lustre: server umount lustre-MDT0000 complete [ 9702.369262] LustreError: 365931:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.375834] LustreError: 365931:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9703.121881] LustreError: 365911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192853 with bad export cookie 4087527702867189324 [ 9703.129068] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.130933] LustreError: 365911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9703.299495] Lustre: server umount lustre-MDT0001 complete [ 9708.774280] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 9710.303454] Key type lgssc unregistered [ 9710.456557] LNet: 369460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9710.461180] LNetError: 369460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9710.470190] LNet: Removed LNI 192.168.203.110@tcp [ 9710.917978] Key type .llcrypt unregistered [ 9710.919276] Key type ._llcrypt unregistered [ 9724.760473] Key type ._llcrypt registered [ 9724.762222] Key type .llcrypt registered [ 9724.817165] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 9725.337262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9725.381547] alg: No test for adler32 (adler32-zlib) [ 9726.304767] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9726.441105] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 9728.080267] Key type lgssc registered [ 9728.707426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9732.203546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9732.213714] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9732.218522] LustreError: 370877:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9732.229810] LustreError: 370877:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9732.233891] LustreError: 370877:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9732.237121] LustreError: 370877:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9732.240485] LustreError: 370877:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9735.673145] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9735.676198] LustreError: 371274:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9735.685832] LustreError: 371274:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9735.690283] LustreError: 371274:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9735.694136] LustreError: 371274:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9735.698711] LustreError: 371274:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9744.211288] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 9745.751174] Key type lgssc unregistered [ 9745.922132] LNet: 372279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9745.926351] LNetError: 372279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9745.936225] LNet: Removed LNI 192.168.203.110@tcp [ 9746.375466] Key type .llcrypt unregistered [ 9746.377129] Key type ._llcrypt unregistered [ 9755.647092] Key type ._llcrypt registered [ 9755.648686] Key type .llcrypt registered [ 9755.699866] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [ 9763.498941] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 9763.843854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9763.851700] alg: No test for adler32 (adler32-zlib) [ 9764.731111] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9764.829797] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 9766.416278] Key type lgssc registered [ 9766.977150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9781.122726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9781.135813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.246038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9782.260123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9782.298356] Lustre: lustre-MDT0000: new disk, initializing [ 9782.331967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9782.342080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9784.022852] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9789.324513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9789.375476] Lustre: 375408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9789.407572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9789.411366] Lustre: Skipped 1 previous similar message [ 9789.451355] Lustre: lustre-MDT0001: new disk, initializing [ 9789.495641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9789.513178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9789.518106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9791.235046] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9795.443830] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9797.769952] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9800.414716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9800.529700] Lustre: lustre-OST0000: new disk, initializing [ 9800.532051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9800.536337] Lustre: 377378:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9800.571356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9802.764180] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9805.828282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9805.832292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9805.877940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9806.912230] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9808.529769] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9808.613186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9810.222993] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9810.315366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9816.036559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.037065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.038884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.047821] Lustre: Skipped 1 previous similar message [ 9817.257442] Lustre: server umount lustre-OST0000 complete [ 9820.129339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9820.133215] LustreError: Skipped 1 previous similar message [ 9820.135770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.142833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9820.146234] Lustre: Skipped 1 previous similar message [ 9821.155063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9821.156727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.079651] Lustre: server umount lustre-MDT0000 complete [ 9826.692773] LustreError: 375401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192977 with bad export cookie 11028010795696568888 [ 9826.696061] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.700281] LustreError: 375401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9826.849264] Lustre: server umount lustre-MDT0001 complete [ 9830.474413] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9831.167058] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:43:01 (1786192981) [ 9834.073266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.291069] LustreError: 379833:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.301456] LustreError: 379833:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9834.319675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.079749] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9839.586511] LustreError: 379834:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.599630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.745647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9841.252138] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9844.037258] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9846.235310] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9848.819960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9848.973254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9851.410349] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9854.745793] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9857.653467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9857.739441] Lustre: lustre-OST0001: new disk, initializing [ 9857.742573] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9857.746418] Lustre: 382668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9860.230320] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9860.826354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9860.832432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9860.879191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9864.812376] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9871.329088] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9871.332654] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9871.332747] LustreError: Skipped 1 previous similar message [ 9871.337795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9871.340693] Lustre: Skipped 1 previous similar message [ 9871.347401] Lustre: Skipped 2 previous similar messages [ 9872.979328] Lustre: server umount lustre-OST0001 complete [ 9876.451912] LustreError: 383102:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9876.459078] LustreError: 383102:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9881.575597] LustreError: 381650:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9881.585543] LustreError: 381650:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9886.177069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9886.180605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.187639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9886.190439] Lustre: Skipped 1 previous similar message [ 9886.693332] LustreError: 381650:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.700945] LustreError: 381650:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9896.932842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9896.933689] LustreError: 383102:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.935950] Lustre: Skipped 4 previous similar messages [ 9896.946270] LustreError: 383102:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9900.001225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9900.094989] Lustre: server umount lustre-OST0000 complete [ 9902.050301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.059528] Lustre: Skipped 2 previous similar messages [ 9916.384464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9916.468919] Lustre: server umount lustre-MDT0000 complete [ 9917.410116] LustreError: 379834:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.425507] LustreError: 379834:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9918.281558] LustreError: 379815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193068 with bad export cookie 11028010795696569742 [ 9918.281940] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9918.287021] LustreError: 379815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9924.489713] Lustre: server umount lustre-MDT0001 complete [ 9929.165336] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [ 9930.493021] Key type lgssc unregistered [ 9930.661736] LNet: 384655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9930.666472] LNetError: 384655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9930.677129] LNet: Removed LNI 192.168.203.110@tcp [ 9931.105279] Key type .llcrypt unregistered [ 9931.106760] Key type ._llcrypt unregistered [ 9955.003780] Key type ._llcrypt registered [ 9955.004872] Key type .llcrypt registered [ 9955.049081] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [ 9955.536589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9955.547332] alg: No test for adler32 (adler32-zlib) [ 9956.432886] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9956.539256] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [ 9958.128156] Key type lgssc registered [ 9958.554237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9962.137256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9962.145644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.268636] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9963.279613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9963.345997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.120340] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9968.662443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.720963] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9968.737223] Lustre: 387429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9968.853703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9970.720276] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9974.014326] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9976.418292] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9979.361542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9979.364465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.369893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9980.385663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9980.386041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9983.513071] Lustre: server umount lustre-MDT0000 complete [ 9984.984881] LustreError: 388171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193135 with bad export cookie 1853370690295705432 [ 9984.988772] LustreError: 388171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9984.988798] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9985.108157] Lustre: server umount lustre-MDT0001 complete [ 9988.969674] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:45:39 (1786193139) [ 9991.853569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.080485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9993.627455] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [ 9996.954029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.096880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.573900] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10001.271487] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10003.530608] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10006.275360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10006.377812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10006.470575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10008.759284] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10011.833623] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10015.641192] Lustre: Permanently deactivating lustre-OST0000 [10015.646304] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10022.396779] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10022.981810] LustreError: 392971:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10022.986348] LustreError: 392971:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10022.989507] LustreError: 392971:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10023.792479] LustreError: 393019:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10023.795386] LustreError: 393019:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10030.615882] Lustre: server umount lustre-OST0000 complete [10031.654652] LustreError: 393269:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10031.658121] LustreError: 393269:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10032.609068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.609583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10032.616503] Lustre: Skipped 1 previous similar message [10032.619221] Lustre: Skipped 1 previous similar message [10037.729615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.733231] Lustre: Skipped 2 previous similar messages [10042.855617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10042.860397] Lustre: Skipped 1 previous similar message [10042.898109] Lustre: server umount lustre-MDT0000 complete [10044.439540] LustreError: 389920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193195 with bad export cookie 1853370690295705992 [10044.442712] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.444451] LustreError: 389920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10054.692884] Lustre: server umount lustre-MDT0001 complete [10057.852024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.198363] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10060.185545] LustreError: 394427:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10062.227716] Lustre: 394765:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.110@tcp [10062.977996] Lustre: server umount MGS complete [10066.878377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10067.027149] LustreError: 395488:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.034089] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10067.035353] LustreError: 395488:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [10067.038162] Lustre: Skipped 1 previous similar message [10067.057527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10068.428740] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10071.372588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.475772] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10072.956911] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10075.780436] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10077.721718] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10078.380526] Lustre: Permanently reactivating lustre-OST0000 [10078.387042] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10078.389848] Lustre: Skipped 2 previous similar messages [10100.704191] LustreError: 397170:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10103.689394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.844685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10103.849274] Lustre: Skipped 1 previous similar message [10106.136690] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10108.888617] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10121.186780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10121.187037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.189515] LustreError: Skipped 1 previous similar message [10121.195389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.198373] Lustre: Skipped 1 previous similar message [10122.769769] Lustre: server umount lustre-OST0000 complete [10126.299891] Lustre: server umount lustre-MDT0000 complete [10126.306215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10126.313406] Lustre: Skipped 1 previous similar message [10126.315832] LustreError: 398943:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.323844] LustreError: 398943:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [10127.578069] LustreError: 395468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193278 with bad export cookie 1853370690295707035 [10127.581244] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10127.584353] LustreError: 395468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10127.703644] Lustre: server umount lustre-MDT0001 complete [10131.484339] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10132.634889] Key type lgssc unregistered [10132.777463] LNet: 400019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10132.780580] LNetError: 400019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10132.790323] LNet: Removed LNI 192.168.203.110@tcp [10133.102022] Key type .llcrypt unregistered [10133.103082] Key type ._llcrypt unregistered [10139.459281] Key type ._llcrypt registered [10139.460318] Key type .llcrypt registered [10139.499795] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [10146.040512] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10146.469815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10146.479575] alg: No test for adler32 (adler32-zlib) [10147.362239] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10147.465923] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10149.056147] Key type lgssc registered [10149.483964] Lustre: Echo OBD driver; http://www.lustre.org/ [10160.950368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10160.958710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.040983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10162.051494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10162.083614] Lustre: lustre-MDT0000: new disk, initializing [10162.101508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10162.107629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10163.539665] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10167.787731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10167.829889] Lustre: 403147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10167.855658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10167.858019] Lustre: Skipped 1 previous similar message [10167.893834] Lustre: lustre-MDT0001: new disk, initializing [10167.927872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10167.941813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10167.946068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10169.409374] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10173.600371] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10175.786294] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10178.365314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10178.500535] Lustre: lustre-OST0000: new disk, initializing [10178.503220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10178.507084] Lustre: 405114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10178.538508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10180.784623] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10184.977535] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10186.243718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10186.249308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10186.284727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10186.601428] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10186.697577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10188.329489] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10188.425927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10191.329548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10191.331389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.333025] LustreError: Skipped 1 previous similar message [10191.338925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10191.343826] Lustre: Skipped 1 previous similar message [10195.319702] Lustre: server umount lustre-OST0000 complete [10199.008960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10199.011748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.017090] Lustre: Skipped 1 previous similar message [10199.019438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10201.570199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10201.572192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10201.580290] Lustre: Skipped 1 previous similar message [10206.689683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10206.693551] Lustre: Skipped 1 previous similar message [10211.296125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10211.357452] Lustre: server umount lustre-MDT0000 complete [10212.723381] LustreError: 403141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193363 with bad export cookie 14462148036249774577 [10212.726560] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10212.728551] LustreError: 403141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10212.848667] Lustre: server umount lustre-MDT0001 complete [10215.889457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10216.493616] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:49:27 (1786193367) [10219.029567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.204088] LustreError: 407568:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.210657] LustreError: 407568:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [10219.230333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10220.740097] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10223.869459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.013886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10225.483891] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10228.182947] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10230.223703] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10232.637367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10232.753965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10234.829341] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10237.372974] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10238.104087] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10248.959228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10252.260677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10252.268790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10256.622727] Lustre: server umount lustre-OST0000 complete [10259.936686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10259.940297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.945379] Lustre: Skipped 1 previous similar message [10262.497468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10262.499428] Lustre: Skipped 3 previous similar messages [10264.154141] Lustre: server umount lustre-MDT0000 complete [10265.504401] LustreError: 410271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193416 with bad export cookie 14462148036249775459 [10265.507747] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10265.509363] LustreError: 410271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10265.628337] Lustre: server umount lustre-MDT0001 complete [10269.479071] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10270.556844] Key type lgssc unregistered [10270.683514] LNet: 411311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10270.686680] LNetError: 411311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10270.695374] LNet: Removed LNI 192.168.203.110@tcp [10270.983963] Key type .llcrypt unregistered [10270.985195] Key type ._llcrypt unregistered [10278.411315] Key type ._llcrypt registered [10278.412776] Key type .llcrypt registered [10278.451767] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10278.560284] Key type .llcrypt unregistered [10278.562197] Key type ._llcrypt unregistered [10285.736491] Key type ._llcrypt registered [10285.739180] Key type .llcrypt registered [10285.781839] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10286.118864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10286.148104] alg: No test for adler32 (adler32-zlib) [10287.041432] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10287.150951] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10288.744149] Key type lgssc registered [10289.261392] Lustre: Echo OBD driver; http://www.lustre.org/ [10292.491542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10292.499908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.690223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10295.276376] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10298.841071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10298.999838] LustreError: 413497:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10299.007879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10299.011086] LustreError: 413497:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [10301.384922] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10304.099122] LustreError: 413503:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.107090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10304.110112] LustreError: 413503:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [10304.364843] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10307.146552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.302774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10308.893589] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10317.795947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.797507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10317.801358] Lustre: Skipped 1 previous similar message [10322.271561] Lustre: server umount lustre-OST0000 complete [10327.010261] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10327.013077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.017989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10327.020369] Lustre: Skipped 1 previous similar message [10328.033115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10328.033438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.880198] Lustre: server umount lustre-MDT0000 complete [10331.077206] LustreError: 413484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193481 with bad export cookie 5892346604670042430 [10331.080252] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10331.082916] LustreError: 413484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10331.201894] Lustre: server umount lustre-MDT0001 complete [10334.941367] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10336.050568] Key type lgssc unregistered [10336.174335] LNet: 416650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10336.178065] LNetError: 416650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10336.185537] LNet: Removed LNI 192.168.203.110@tcp [10336.500734] Key type .llcrypt unregistered [10336.502266] Key type ._llcrypt unregistered [10346.231682] Key type ._llcrypt registered [10346.233487] Key type .llcrypt registered [10346.282515] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10346.741192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10346.807240] alg: No test for adler32 (adler32-zlib) [10347.670636] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10347.770342] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10349.360134] Key type lgssc registered [10349.729814] Lustre: Echo OBD driver; http://www.lustre.org/ [10352.605482] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10352.614109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10368.032356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10373.152311] LustreError: 418019:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.110@tcp: failed processing log, type 4: rc = -110 [10399.776196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10401.768109] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10404.314291] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10406.767628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.949993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.979363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10408.518879] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10411.713468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.847602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10413.310719] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10415.928694] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10417.917811] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10422.242291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10422.242491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10422.246234] LustreError: Skipped 1 previous similar message [10422.251885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10422.255695] Lustre: Skipped 1 previous similar message [10425.318501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10425.321293] Lustre: Skipped 1 previous similar message [10426.322590] Lustre: server umount lustre-OST0000 complete [10430.433800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.434783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.438702] Lustre: Skipped 2 previous similar messages [10433.883651] Lustre: server umount lustre-MDT0000 complete [10435.192544] LustreError: 418026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193585 with bad export cookie 9090458400829651091 [10435.195580] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10435.196916] LustreError: 418026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10435.321112] Lustre: server umount lustre-MDT0001 complete [10439.030146] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10440.151412] Key type lgssc unregistered [10440.288337] LNet: 421593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10440.291986] LNetError: 421593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10440.304473] LNet: Removed LNI 192.168.203.110@tcp [10440.689768] Key type .llcrypt unregistered [10440.691673] Key type ._llcrypt unregistered [10450.554669] Key type ._llcrypt registered [10450.556164] Key type .llcrypt registered [10450.599099] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10451.126155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10451.154536] alg: No test for adler32 (adler32-zlib) [10452.036743] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10452.144963] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10453.728172] Key type lgssc registered [10454.191633] Lustre: Echo OBD driver; http://www.lustre.org/ [10457.365289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10457.374188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.567331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10460.201621] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10463.404994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.548031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10465.058678] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10467.907705] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10470.042770] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10472.645534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10472.782558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10474.926860] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10477.753136] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10484.072140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10489.316305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.317497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10489.318900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.321498] Lustre: Skipped 2 previous similar messages [10491.324990] Lustre: server umount lustre-MDT0000 complete [10492.382086] LustreError: 422966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.10@tcp arrived at 1786193643 with bad export cookie 11508341231984621407 [10492.390023] LustreError: 422966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10493.169397] LustreError: 422968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193643 with bad export cookie 11508341231984620756 [10493.174535] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.433632] LustreError: 425476:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.434366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.434776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10494.442203] LustreError: 425476:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [10494.447988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10494.457925] Lustre: Skipped 3 previous similar messages [10498.465109] LustreError: 422987:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.465834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10498.472868] LustreError: 422987:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [10499.285537] Lustre: server umount lustre-OST0000 complete [10501.707687] Lustre: server umount lustre-MDT0001 complete [10505.467839] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10506.599683] Key type lgssc unregistered [10506.742698] LNet: 426673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10506.749036] LNetError: 426673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10506.756953] LNet: Removed LNI 192.168.203.110@tcp [10507.110956] Key type .llcrypt unregistered [10507.112334] Key type ._llcrypt unregistered [10516.766525] Key type ._llcrypt registered [10516.767687] Key type .llcrypt registered [10516.810044] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10517.263471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10517.269828] alg: No test for adler32 (adler32-zlib) [10518.131116] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10518.221181] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10519.808159] Key type lgssc registered [10520.192231] Lustre: Echo OBD driver; http://www.lustre.org/ [10522.878492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10522.886115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10524.028892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10525.364210] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10528.113674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.238141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10529.553106] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10531.990099] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10533.847853] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10536.195867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10536.318114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10538.411413] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10540.992368] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10548.646946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10553.828161] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10553.828865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.833274] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10553.838132] Lustre: Skipped 1 previous similar message [10553.842525] Lustre: Skipped 1 previous similar message [10556.601073] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.10@tcp (stopping) [10558.616477] Lustre: server umount lustre-MDT0001 complete [10558.946071] LustreError: 428066:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.953232] LustreError: 428066:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [10564.065921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10564.066873] LustreError: 428792:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.068609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.079634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10566.423269] Lustre: server umount lustre-OST0000 complete [10567.714369] Lustre: server umount lustre-MDT0000 complete [10572.106746] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10573.128968] Key type lgssc unregistered [10573.259605] LNet: 431770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10573.263206] LNetError: 431770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10573.272475] LNet: Removed LNI 192.168.203.110@tcp [10573.604514] Key type .llcrypt unregistered [10573.605560] Key type ._llcrypt unregistered [10581.977181] Key type ._llcrypt registered [10581.978829] Key type .llcrypt registered [10582.026413] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [10588.342325] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10588.699755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10588.756032] alg: No test for adler32 (adler32-zlib) [10589.642424] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10589.739087] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10591.320152] Key type lgssc registered [10591.742502] Lustre: Echo OBD driver; http://www.lustre.org/ [10604.691173] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10610.320026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10610.325969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.418103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10611.429837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10611.468077] Lustre: lustre-MDT0000: new disk, initializing [10611.492320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10611.500166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10613.028668] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10619.160619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10619.269785] Lustre: lustre-OST0000: new disk, initializing [10619.272153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10619.274812] Lustre: Skipped 1 previous similar message [10619.277627] Lustre: 436776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10619.305899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10621.483172] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10625.503341] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10625.553614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10625.560398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10625.572093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10631.072390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.128786] Lustre: 435756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10631.162476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10631.200884] Lustre: lustre-MDT0001: new disk, initializing [10631.246951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10631.261670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10631.264527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10632.543177] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10636.326370] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10636.412975] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10637.889676] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10637.975175] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10642.916958] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.917770] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10642.922692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10642.925811] Lustre: Skipped 1 previous similar message [10645.714429] Lustre: server umount lustre-MDT0001 complete [10653.155800] LustreError: 438362:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10653.162141] LustreError: 438362:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [10658.274194] LustreError: 435768:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10658.274413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.285189] Lustre: Skipped 1 previous similar message [10658.287112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10658.289186] Lustre: Skipped 1 previous similar message [10663.393164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10663.393799] LustreError: 435767:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.344047] Lustre: server umount lustre-OST0000 complete [10671.866207] Lustre: server umount lustre-MDT0000 complete [10676.303606] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10677.393233] Key type lgssc unregistered [10677.523542] LNet: 440521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10677.527137] LNetError: 440521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10677.538269] LNet: Removed LNI 192.168.203.110@tcp [10677.861513] Key type .llcrypt unregistered [10677.862747] Key type ._llcrypt unregistered [10692.048568] Key type ._llcrypt registered [10692.049788] Key type .llcrypt registered [10692.089456] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [10692.210462] Key type .llcrypt unregistered [10692.211670] Key type ._llcrypt unregistered [10698.877668] Key type ._llcrypt registered [10698.879161] Key type .llcrypt registered [10698.919353] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [10699.264662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10699.309266] alg: No test for adler32 (adler32-zlib) [10700.166964] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10700.249182] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [10701.832208] Key type lgssc registered [10702.229429] Lustre: Echo OBD driver; http://www.lustre.org/ [10711.462410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10711.468410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.555951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10712.566305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10712.602932] Lustre: lustre-MDT0000: new disk, initializing [10712.627383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10712.636032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10714.056429] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10718.532833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10718.578905] Lustre: 443874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10718.608481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10718.611785] Lustre: Skipped 1 previous similar message [10718.649603] Lustre: lustre-MDT0001: new disk, initializing [10718.695916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10718.712462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10718.718400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10720.299275] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10724.466331] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10726.648674] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10729.271501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10729.380035] Lustre: lustre-OST0000: new disk, initializing [10729.381908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10729.384889] Lustre: 445843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10729.414810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10731.536949] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10731.651476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10731.655992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10731.693505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10735.386742] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10739.168879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10739.172403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.178828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10742.242061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10742.242586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.836582] Lustre: server umount lustre-MDT0000 complete [10745.084070] LustreError: 443867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193895 with bad export cookie 7612022068328429583 [10745.087515] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10745.088581] LustreError: 443867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10745.201885] Lustre: server umount lustre-MDT0001 complete [10756.776888] Lustre: server umount lustre-OST0000 complete [10760.295956] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:58:31 (1786193911) [10762.845528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10763.021079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10764.423476] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10767.254135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.360262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10768.606424] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10770.914297] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10772.626248] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10787.814302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10787.878297] Lustre: Found index 0 for lustre-OST0000, updating log [10787.937533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10788.967318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10789.734585] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [10792.078949] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10794.464878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10794.468607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10794.474572] Lustre: Skipped 1 previous similar message [10794.477347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10794.480293] Lustre: Skipped 2 previous similar messages [10799.311160] Lustre: server umount lustre-OST0000 complete [10813.408763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10813.411223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.415302] Lustre: Skipped 1 previous similar message [10813.417132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.418950] Lustre: Skipped 3 previous similar messages [10818.135920] Lustre: server umount lustre-MDT0000 complete [10819.370229] LustreError: 447848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193970 with bad export cookie 7612022068328430962 [10819.373612] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10819.374261] LustreError: 447848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10819.483940] Lustre: server umount lustre-MDT0001 complete [10822.162925] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:59:33 (1786193973) [10849.760301] LNet: 1 local NIs in recovery (showing 1): 192.168.203.110@tcp [11401.953850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.105606] LustreError: 454279:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.114100] LustreError: 454279:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [11402.131534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.531223] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11406.494191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.008609] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11410.356868] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11412.179456] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11427.427051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11427.517510] Lustre: Found index 0 for lustre-OST0000, updating log [11429.585888] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11429.673290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11431.998131] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11433.699355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.703685] Lustre: Skipped 1 previous similar message [11433.705553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11433.708046] Lustre: Skipped 2 previous similar messages [11438.797808] Lustre: server umount lustre-OST0000 complete [11440.027142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11440.027952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11440.030823] Lustre: Skipped 1 previous similar message [11440.036162] Lustre: Skipped 1 previous similar message [11441.057177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11441.059127] Lustre: Skipped 1 previous similar message [11443.105320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11443.108150] Lustre: Skipped 3 previous similar messages [11446.167507] Lustre: server umount lustre-MDT0000 complete [11446.178603] LustreError: 454298:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.187213] LustreError: 454298:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [11447.201221] LustreError: 454280:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.206638] LustreError: 454280:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [11447.378053] LustreError: 457126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194598 with bad export cookie 7612022068328432026 [11447.381465] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11447.381543] LustreError: 457126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11447.505100] Lustre: server umount lustre-MDT0001 complete [11463.136273] LNet: 1 local NIs in recovery (showing 1): 192.168.203.110@tcp [11464.754015] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [11467.684975] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11478.644757] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:10:29 (1786194629) [11484.277098] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [11485.259468] Key type lgssc unregistered [11490.528429] LNet: 460932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11490.531196] LNetError: 460932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11556.070313] LNet: Removed LNI 192.168.203.110@tcp [11556.368249] Key type .llcrypt unregistered [11556.369531] Key type ._llcrypt unregistered [11563.074906] Key type ._llcrypt registered [11563.076068] Key type .llcrypt registered [11563.114770] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11563.474784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11563.534919] alg: No test for adler32 (adler32-zlib) [11564.390638] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11564.484072] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [11566.064151] Key type lgssc registered [11566.445527] Lustre: Echo OBD driver; http://www.lustre.org/ [11567.957187] LNet: Added LNI 192.168.203.110@tcp100 [8/256/0/180] [11568.255654] LNet: Added LNI 192.168.203.110@tcp101 [8/256/0/180] [11568.548216] LNet: Added LNI 192.168.203.110@tcp102 [8/256/0/180] [11568.836721] LNet: Added LNI 192.168.203.110@tcp103 [8/256/0/180] [11569.119507] LNet: Added LNI 192.168.203.110@tcp104 [8/256/0/180] [11569.413770] LNet: Added LNI 192.168.203.110@tcp105 [8/256/0/180] [11569.716408] LNet: Added LNI 192.168.203.110@tcp106 [8/256/0/180] [11570.012268] LNet: Added LNI 192.168.203.110@tcp107 [8/256/0/180] [11570.302134] LNet: Added LNI 192.168.203.110@tcp108 [8/256/0/180] [11570.598243] LNet: Added LNI 192.168.203.110@tcp109 [8/256/0/180] [11570.896575] LNet: Added LNI 192.168.203.110@tcp110 [8/256/0/180] [11571.183957] LNet: Added LNI 192.168.203.110@tcp111 [8/256/0/180] [11571.468539] LNet: Added LNI 192.168.203.110@tcp112 [8/256/0/180] [11571.765977] LNet: Added LNI 192.168.203.110@tcp113 [8/256/0/180] [11572.059437] LNet: Added LNI 192.168.203.110@tcp114 [8/256/0/180] [11572.354072] LNet: Added LNI 192.168.203.110@tcp115 [8/256/0/180] [11572.654357] LNet: Added LNI 192.168.203.110@tcp116 [8/256/0/180] [11572.976350] LNet: Added LNI 192.168.203.110@tcp117 [8/256/0/180] [11573.275939] LNet: Added LNI 192.168.203.110@tcp118 [8/256/0/180] [11573.571865] LNet: Added LNI 192.168.203.110@tcp119 [8/256/0/180] [11573.892285] LNet: Added LNI 192.168.203.110@tcp120 [8/256/0/180] [11574.210477] LNet: Added LNI 192.168.203.110@tcp121 [8/256/0/180] [11574.509559] LNet: Added LNI 192.168.203.110@tcp122 [8/256/0/180] [11574.819455] LNet: Added LNI 192.168.203.110@tcp123 [8/256/0/180] [11575.130186] LNet: Added LNI 192.168.203.110@tcp124 [8/256/0/180] [11575.435805] LNet: Added LNI 192.168.203.110@tcp125 [8/256/0/180] [11575.735929] LNet: Added LNI 192.168.203.110@tcp126 [8/256/0/180] [11576.052022] LNet: Added LNI 192.168.203.110@tcp127 [8/256/0/180] [11576.360493] LNet: Added LNI 192.168.203.110@tcp128 [8/256/0/180] [11576.669229] LNet: Added LNI 192.168.203.110@tcp129 [8/256/0/180] [11576.974309] LNet: Added LNI 192.168.203.110@tcp130 [8/256/0/180] [11577.277225] LNet: Added LNI 192.168.203.110@tcp131 [8/256/0/180] [11577.592844] LNet: Added LNI 192.168.203.110@tcp132 [8/256/0/180] [11577.887496] LNet: Added LNI 192.168.203.110@tcp133 [8/256/0/180] [11578.186294] LNet: Added LNI 192.168.203.110@tcp134 [8/256/0/180] [11578.481399] LNet: Added LNI 192.168.203.110@tcp135 [8/256/0/180] [11580.750768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11580.759977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11581.799850] Lustre: 463928:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.110@tcp: can't add NID 192.168.203.110@tcp130: rc = -75 [11581.830326] LNetError: 463948:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11581.833436] LNetError: Protocol error connecting to 192.168.203.110@tcp118 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11581.836964] LNetError: Skipped 3 previous similar messages [11581.863885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11581.873159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11581.909755] Lustre: lustre-MDT0000: new disk, initializing [11581.920091] LNet: 4 local NIs in recovery (showing 4): 192.168.203.110@tcp125, 192.168.203.110@tcp126, 192.168.203.110@tcp127, 192.168.203.110@tcp128 [11581.932816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11581.959628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11582.882607] LNetError: 463957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11582.888157] LNetError: 463957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [11582.891958] LNetError: Protocol error connecting to 192.168.203.110@tcp103 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11582.895537] LNetError: Skipped 7 previous similar messages [11583.277111] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11584.997351] LNetError: 463957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11585.000472] LNetError: 463957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [11585.003126] LNetError: Protocol error connecting to 192.168.203.110@tcp127 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11585.007742] LNetError: Skipped 9 previous similar messages [11587.326190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.381246] Lustre: 463943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11587.404672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11587.407206] Lustre: Skipped 1 previous similar message [11587.446517] Lustre: lustre-MDT0001: new disk, initializing [11587.454779] LNetError: 461395:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11587.458854] LNetError: 461395:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11587.458985] Lustre: 464750:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.110@tcp: can't add NID 192.168.203.110@tcp130: rc = -75 [11587.463139] LNetError: Protocol error connecting to 192.168.203.110@tcp132 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11587.466874] Lustre: 464750:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11587.474031] LNetError: Skipped 2 previous similar messages [11587.511993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11587.524715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11587.529918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11588.885845] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11590.263805] Lustre: 465114:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.110@tcp: can't add NID 192.168.203.110@tcp130: rc = -75 [11590.267372] Lustre: 465114:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11592.596662] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11594.210147] LNetError: 463960:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11594.215395] LNetError: 463960:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [11594.217749] LNetError: Protocol error connecting to 192.168.203.110@tcp132 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11594.222985] LNetError: Skipped 10 previous similar messages [11594.418408] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11596.660061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11596.762874] Lustre: lustre-OST0000: new disk, initializing [11596.764523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11596.766997] Lustre: 465920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11596.798792] Lustre: 465925:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.110@tcp: can't add NID 192.168.203.110@tcp130: rc = -75 [11596.801898] Lustre: 465925:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11596.821768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11598.787710] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11602.401104] LNetError: 463961:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11602.404975] LNetError: 463961:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [11602.407283] LNetError: Protocol error connecting to 192.168.203.110@tcp132 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11602.410139] LNetError: Skipped 3 previous similar messages [11602.492988] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11604.819145] Lustre: server umount lustre-OST0000 complete [11606.511769] Lustre: 467065:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.110@tcp: can't add NID 192.168.203.110@tcp130: rc = -75 [11606.514758] Lustre: 467065:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11606.524610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11606.525657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11606.528417] Lustre: Skipped 1 previous similar message [11606.531612] Lustre: Skipped 1 previous similar message [11611.617295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11611.619245] Lustre: Skipped 1 previous similar message [11616.739777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11616.741870] Lustre: Skipped 1 previous similar message [11618.785077] LNetError: 463957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.110 [11618.788252] LNetError: 463957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [11618.790719] LNetError: Protocol error connecting to 192.168.203.110@tcp132 at host 192.168.203.110:988: is it running a compatible version of Lustre? [11618.793935] LNetError: Skipped 8 previous similar messages [11620.320144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11620.388261] Lustre: server umount lustre-MDT0000 complete [11621.650130] LustreError: 463937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194772 with bad export cookie 8382350608626870160 [11621.653625] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.654793] LustreError: 463937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11621.773304] Lustre: server umount lustre-MDT0001 complete [11625.347229] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [11626.351775] Key type lgssc unregistered [11626.518481] LNet: 467828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11626.521777] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11626.524096] LNet: Removed LNI 192.168.203.110@tcp [11627.552137] LNet: Removed LNI 192.168.203.110@tcp100 [11627.553795] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11627.557098] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11627.560812] LNet: Removed LNI 192.168.203.110@tcp101 [11628.576121] LNet: Removed LNI 192.168.203.110@tcp102 [11628.577689] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11628.580676] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11628.583766] LNet: Removed LNI 192.168.203.110@tcp103 [11629.600151] LNet: Removed LNI 192.168.203.110@tcp104 [11630.624120] LNet: Removed LNI 192.168.203.110@tcp105 [11630.626336] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11630.629884] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11630.633847] LNet: Removed LNI 192.168.203.110@tcp106 [11631.648130] LNet: Removed LNI 192.168.203.110@tcp107 [11632.672126] LNet: Removed LNI 192.168.203.110@tcp108 [11633.696131] LNet: Removed LNI 192.168.203.110@tcp109 [11634.720223] LNet: Removed LNI 192.168.203.110@tcp110 [11634.722685] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11634.726845] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11634.730083] LNet: Removed LNI 192.168.203.110@tcp111 [11635.744145] LNet: Removed LNI 192.168.203.110@tcp112 [11636.768144] LNet: Removed LNI 192.168.203.110@tcp113 [11637.792127] LNet: Removed LNI 192.168.203.110@tcp114 [11638.816135] LNet: Removed LNI 192.168.203.110@tcp115 [11639.840121] LNet: Removed LNI 192.168.203.110@tcp116 [11640.864171] LNet: Removed LNI 192.168.203.110@tcp117 [11641.888188] LNet: Removed LNI 192.168.203.110@tcp118 [11642.912167] LNet: Removed LNI 192.168.203.110@tcp119 [11642.914102] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11642.916193] LNetError: 467828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11642.918346] LNet: Removed LNI 192.168.203.110@tcp120 [11643.936151] LNet: Removed LNI 192.168.203.110@tcp121 [11644.960116] LNet: Removed LNI 192.168.203.110@tcp122 [11645.984104] LNet: Removed LNI 192.168.203.110@tcp123 [11647.008143] LNet: Removed LNI 192.168.203.110@tcp124 [11648.032082] LNet: Removed LNI 192.168.203.110@tcp125 [11649.056156] LNet: Removed LNI 192.168.203.110@tcp126 [11649.057603] LNet: Removed LNI 192.168.203.110@tcp127 [11650.080109] LNet: Removed LNI 192.168.203.110@tcp128 [11651.104177] LNet: Removed LNI 192.168.203.110@tcp129 [11652.128123] LNet: Removed LNI 192.168.203.110@tcp130 [11653.152156] LNet: Removed LNI 192.168.203.110@tcp131 [11653.153445] LNet: Removed LNI 192.168.203.110@tcp132 [11654.176108] LNet: Removed LNI 192.168.203.110@tcp133 [11655.200108] LNet: Removed LNI 192.168.203.110@tcp134 [11656.230422] LNet: Removed LNI 192.168.203.110@tcp135 [11656.514264] Key type .llcrypt unregistered [11656.515914] Key type ._llcrypt unregistered [11663.279084] Key type ._llcrypt registered [11663.280113] Key type .llcrypt registered [11663.316050] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [11668.992838] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11669.419116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11669.433712] alg: No test for adler32 (adler32-zlib) [11670.283126] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11670.361335] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [11671.936152] Key type lgssc registered [11672.287882] Lustre: Echo OBD driver; http://www.lustre.org/ [11682.913964] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:13:53 (1786194833) [11688.202250] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [11689.096342] Key type lgssc unregistered [11689.219224] LNet: 471890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11689.221357] LNetError: 471890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11689.229186] LNet: Removed LNI 192.168.203.110@tcp [11689.492353] Key type .llcrypt unregistered [11689.493314] Key type ._llcrypt unregistered [11695.572509] Key type ._llcrypt registered [11695.573490] Key type .llcrypt registered [11695.608749] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11696.036672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11696.046825] alg: No test for adler32 (adler32-zlib) [11696.894542] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11696.977688] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [11698.552151] Key type lgssc registered [11698.906968] Lustre: Echo OBD driver; http://www.lustre.org/ [11701.780798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11701.785832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11702.857963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11702.867584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11702.902863] Lustre: lustre-MDT0000: new disk, initializing [11702.923621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11702.929858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11704.174973] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11708.007919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.044337] Lustre: 473095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11708.064913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11708.067143] Lustre: Skipped 1 previous similar message [11708.099492] Lustre: lustre-MDT0001: new disk, initializing [11708.143403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11708.153218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11708.156354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11709.332171] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11713.046989] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11714.904242] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11717.188341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11717.290382] Lustre: lustre-OST0000: new disk, initializing [11717.292429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11717.295732] Lustre: 475063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11717.328734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11719.163215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11719.166878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11719.202078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11719.358243] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11723.062567] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11724.303795] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing lnet_if_list [11726.026197] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11726.039426] LNet: Added LNI 192.168.203.110@tcp10 [8/256/0/180] [11727.882241] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11727.897028] LNet: Added LNI 192.168.203.110@tcp11 [8/256/0/180] [11729.242306] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11729.254909] LNet: Added LNI 192.168.203.110@tcp12 [8/256/0/180] [11730.685620] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11730.699033] LNet: Added LNI 192.168.203.110@tcp13 [8/256/0/180] [11732.154376] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11732.165849] LNet: Added LNI 192.168.203.110@tcp14 [8/256/0/180] [11733.615663] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11733.629550] LNet: Added LNI 192.168.203.110@tcp15 [8/256/0/180] [11735.152442] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11735.168400] LNet: Added LNI 192.168.203.110@tcp16 [8/256/0/180] [11736.681387] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11736.691835] LNet: Added LNI 192.168.203.110@tcp17 [8/256/0/180] [11738.188714] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11738.199965] LNet: Added LNI 192.168.203.110@tcp18 [8/256/0/180] [11739.745917] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11739.758386] LNet: Added LNI 192.168.203.110@tcp19 [8/256/0/180] [11741.430769] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11741.444060] LNet: Added LNI 192.168.203.110@tcp20 [8/256/0/180] [11743.172627] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11743.185482] LNet: Added LNI 192.168.203.110@tcp21 [8/256/0/180] [11744.908288] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11744.919990] LNet: Added LNI 192.168.203.110@tcp22 [8/256/0/180] [11746.713939] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11746.728219] LNet: Added LNI 192.168.203.110@tcp23 [8/256/0/180] [11748.502744] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11748.514862] LNet: Added LNI 192.168.203.110@tcp24 [8/256/0/180] [11750.373953] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11750.384163] LNet: Added LNI 192.168.203.110@tcp25 [8/256/0/180] [11752.363879] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11752.381526] LNet: Added LNI 192.168.203.110@tcp26 [8/256/0/180] [11754.335887] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11754.349614] LNet: Added LNI 192.168.203.110@tcp27 [8/256/0/180] [11756.208204] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11756.222660] LNet: Added LNI 192.168.203.110@tcp28 [8/256/0/180] [11758.129018] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11758.143173] LNet: Added LNI 192.168.203.110@tcp29 [8/256/0/180] [11760.174102] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11760.186400] LNet: Added LNI 192.168.203.110@tcp30 [8/256/0/180] [11762.250064] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11762.259642] LNet: Added LNI 192.168.203.110@tcp31 [8/256/0/180] [11764.462241] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11764.476691] LNet: Added LNI 192.168.203.110@tcp32 [8/256/0/180] [11766.608209] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11766.622295] LNet: Added LNI 192.168.203.110@tcp33 [8/256/0/180] [11768.707868] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11768.721487] LNet: Added LNI 192.168.203.110@tcp34 [8/256/0/180] [11770.908383] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11770.922688] LNet: Added LNI 192.168.203.110@tcp35 [8/256/0/180] [11773.132708] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11773.142553] LNet: Added LNI 192.168.203.110@tcp36 [8/256/0/180] [11775.349470] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11775.362898] LNet: Added LNI 192.168.203.110@tcp37 [8/256/0/180] [11777.724650] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11777.738075] LNet: Added LNI 192.168.203.110@tcp38 [8/256/0/180] [11779.951454] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11779.962299] LNet: Added LNI 192.168.203.110@tcp39 [8/256/0/180] [11782.235176] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11782.248709] LNet: Added LNI 192.168.203.110@tcp40 [8/256/0/180] [11784.533768] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11784.544894] LNet: Added LNI 192.168.203.110@tcp41 [8/256/0/180] [11786.922015] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11786.932798] LNet: Added LNI 192.168.203.110@tcp42 [8/256/0/180] [11789.539311] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11789.551877] LNet: Added LNI 192.168.203.110@tcp43 [8/256/0/180] [11792.083838] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11792.096048] LNet: Added LNI 192.168.203.110@tcp44 [8/256/0/180] [11794.499115] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11794.508730] LNet: Added LNI 192.168.203.110@tcp45 [8/256/0/180] [11796.923194] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11796.934083] LNet: Added LNI 192.168.203.110@tcp46 [8/256/0/180] [11799.404266] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11799.414853] LNet: Added LNI 192.168.203.110@tcp47 [8/256/0/180] [11801.909808] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11801.920749] LNet: Added LNI 192.168.203.110@tcp48 [8/256/0/180] [11804.359490] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11804.369283] LNet: Added LNI 192.168.203.110@tcp49 [8/256/0/180] [11806.967387] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11806.979130] LNet: Added LNI 192.168.203.110@tcp50 [8/256/0/180] [11815.904914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11815.907841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.912879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.417463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.419351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.420948] Lustre: Skipped 1 previous similar message [11816.428571] Lustre: Skipped 1 previous similar message [11820.244881] Lustre: server umount lustre-MDT0000 complete [11821.407096] LustreError: 473086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194972 with bad export cookie 16235873979236411652 [11821.410438] LustreError: 473086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11821.410540] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11821.521211] Lustre: server umount lustre-MDT0001 complete [11828.458483] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11831.330406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11831.776277] LustreError: 472366:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e740158e x1872961047969280/t0(0) o250->MGC192.168.203.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11831.880507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11833.239479] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11836.017029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.108462] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.140194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11837.158436] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11837.448384] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11838.176098] Lustre: 472367:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786194972/real 1786194972] req@0000000058c246b6 x1872961047968896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786194988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11839.758450] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11841.481836] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11843.552121] Lustre: 472369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786194977/real 1786194977] req@00000000c011aeda x1872961047969024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786194993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11848.352097] Lustre: 472370:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786194982/real 1786194982] req@0000000091a125e8 x1872961047969152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786194998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11851.746409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11851.749878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.755249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11851.757317] Lustre: Skipped 1 previous similar message [11854.684569] Lustre: server umount lustre-OST0000 complete [11856.865196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11856.868617] LustreError: Skipped 1 previous similar message [11856.870057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11856.874352] Lustre: Skipped 1 previous similar message [11856.877024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11856.879374] Lustre: Skipped 2 previous similar messages [11862.105651] Lustre: server umount lustre-MDT0000 complete [11863.008923] LustreError: 499251:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.015721] LustreError: 499251:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [11863.340251] LustreError: 473086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195014 with bad export cookie 16235873979236413682 [11863.343674] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.344147] LustreError: 473086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11863.464639] Lustre: server umount lustre-MDT0001 complete [11867.023308] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [11868.057923] Key type lgssc unregistered [11868.210446] LNet: 502025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11868.214060] LNetError: 502025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11868.216901] LNet: Removed LNI 192.168.203.110@tcp [11868.218429] LNet: Removed LNI 192.168.203.110@tcp10 [11868.220144] LNet: Removed LNI 192.168.203.110@tcp11 [11868.222256] LNet: Removed LNI 192.168.203.110@tcp12 [11868.223918] LNet: Removed LNI 192.168.203.110@tcp13 [11868.225502] LNet: Removed LNI 192.168.203.110@tcp14 [11868.226937] LNet: Removed LNI 192.168.203.110@tcp15 [11868.228916] LNet: Removed LNI 192.168.203.110@tcp16 [11868.230974] LNet: Removed LNI 192.168.203.110@tcp17 [11868.232603] LNet: Removed LNI 192.168.203.110@tcp18 [11868.233965] LNet: Removed LNI 192.168.203.110@tcp19 [11868.235366] LNet: Removed LNI 192.168.203.110@tcp20 [11868.236991] LNet: Removed LNI 192.168.203.110@tcp21 [11868.238283] LNet: Removed LNI 192.168.203.110@tcp22 [11868.240196] LNet: Removed LNI 192.168.203.110@tcp23 [11868.242154] LNet: Removed LNI 192.168.203.110@tcp24 [11868.244089] LNet: Removed LNI 192.168.203.110@tcp25 [11868.246057] LNet: Removed LNI 192.168.203.110@tcp26 [11868.247825] LNet: Removed LNI 192.168.203.110@tcp27 [11868.250104] LNet: Removed LNI 192.168.203.110@tcp28 [11868.251865] LNet: Removed LNI 192.168.203.110@tcp29 [11868.253887] LNet: Removed LNI 192.168.203.110@tcp30 [11868.256067] LNet: Removed LNI 192.168.203.110@tcp31 [11868.257544] LNet: Removed LNI 192.168.203.110@tcp32 [11868.259167] LNet: Removed LNI 192.168.203.110@tcp33 [11868.260599] LNet: Removed LNI 192.168.203.110@tcp34 [11868.262383] LNet: Removed LNI 192.168.203.110@tcp35 [11868.263791] LNet: Removed LNI 192.168.203.110@tcp36 [11868.265099] LNet: Removed LNI 192.168.203.110@tcp37 [11868.266317] LNet: Removed LNI 192.168.203.110@tcp38 [11868.267662] LNet: Removed LNI 192.168.203.110@tcp39 [11868.268985] LNet: Removed LNI 192.168.203.110@tcp40 [11868.270685] LNet: Removed LNI 192.168.203.110@tcp41 [11868.271939] LNet: Removed LNI 192.168.203.110@tcp42 [11868.273250] LNet: Removed LNI 192.168.203.110@tcp43 [11868.275081] LNet: Removed LNI 192.168.203.110@tcp44 [11868.276684] LNet: Removed LNI 192.168.203.110@tcp45 [11868.278459] LNet: Removed LNI 192.168.203.110@tcp46 [11868.281018] LNet: Removed LNI 192.168.203.110@tcp47 [11868.283172] LNet: Removed LNI 192.168.203.110@tcp48 [11868.285326] LNet: Removed LNI 192.168.203.110@tcp49 [11868.291314] LNet: Removed LNI 192.168.203.110@tcp50 [11868.607745] Key type .llcrypt unregistered [11868.609088] Key type ._llcrypt unregistered [11874.983797] Key type ._llcrypt registered [11874.984879] Key type .llcrypt registered [11875.020921] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [11881.187582] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11881.629827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11881.640295] alg: No test for adler32 (adler32-zlib) [11882.499798] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11882.594170] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [11884.176142] Key type lgssc registered [11884.566541] Lustre: Echo OBD driver; http://www.lustre.org/ [11895.760772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11895.766900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11896.856395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11896.869953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11896.914263] Lustre: lustre-MDT0000: new disk, initializing [11896.936582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11896.943739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11898.435064] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11902.701669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.745212] Lustre: 505192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11902.771634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11902.774326] Lustre: Skipped 1 previous similar message [11902.810755] Lustre: lustre-MDT0001: new disk, initializing [11902.844613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11902.858167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11902.861243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11904.272518] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11907.882374] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11909.509698] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11911.798634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11911.865323] Lustre: *** cfs_fail_loc=90f, val=0*** [11911.891386] Lustre: lustre-OST0000: new disk, initializing [11911.893048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11911.895548] Lustre: 507211:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11911.922235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11913.462580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11913.467311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11913.496608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11913.645141] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11917.092451] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11923.938325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11923.940278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.941236] LustreError: Skipped 1 previous similar message [11923.946770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11923.949913] Lustre: Skipped 1 previous similar message [11925.265724] Lustre: server umount lustre-OST0000 complete [11928.544962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11928.547569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.551092] Lustre: Skipped 1 previous similar message [11928.552941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11932.634889] Lustre: server umount lustre-MDT0000 complete [11933.891120] LustreError: 507210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195084 with bad export cookie 7461520173186269661 [11933.893635] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11933.895175] LustreError: 507210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11933.999990] Lustre: server umount lustre-MDT0001 complete [11937.442924] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [11938.379919] Key type lgssc unregistered [11938.505343] LNet: 509117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11938.508035] LNetError: 509117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11938.514338] LNet: Removed LNI 192.168.203.110@tcp [11938.802807] Key type .llcrypt unregistered [11938.804137] Key type ._llcrypt unregistered [11945.631166] Key type ._llcrypt registered [11945.632166] Key type .llcrypt registered [11945.663969] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [11945.979219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11946.014084] alg: No test for adler32 (adler32-zlib) [11946.848584] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11946.924681] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [11948.504089] Key type lgssc registered [11948.830885] Lustre: Echo OBD driver; http://www.lustre.org/ [11951.156600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11951.162438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.286424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11953.515094] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11956.052736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11956.165844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11957.325847] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11959.466615] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11961.065339] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11963.092872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11963.203914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11965.120179] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11967.495787] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11981.269415] loop0: detected capacity change from 0 to 204800000 [11981.325723] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11981.329537] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11981.332889] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11983.195350] loop0: detected capacity change from 0 to 204800000 [11983.248638] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11983.253622] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11983.257340] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11984.867826] loop0: detected capacity change from 0 to 204800000 [11986.645768] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11986.691873] Lustre: fs15246-OST0000: new disk, initializing [11986.693628] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11986.697340] Lustre: 513973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11986.714406] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11988.576819] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [11992.297964] loop1: detected capacity change from 0 to 204800000 [11994.088461] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11994.124572] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11994.150433] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11994.182076] Lustre: fs15246-MDT0000: new disk, initializing [11994.235761] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11994.245544] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11994.286393] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11995.296784] Lustre: 515094:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f8d2814e x1872961310064640/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786195161 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11995.455432] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [12002.791970] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12004.384249] LustreError: 515094:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [12004.388533] LustreError: 515094:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [12004.392232] LustreError: 515094:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [12007.906757] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12008.334701] Lustre: server umount fs15246-MDT0000 complete [12011.136470] Lustre: server umount fs15246-OST0000 complete [12017.633166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12017.635789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.640203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12018.148072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.303385] Lustre: server umount lustre-OST0000 complete [12022.752789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12022.755140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.759759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.762545] Lustre: Skipped 2 previous similar messages [12027.616433] Lustre: server umount lustre-MDT0000 complete [12028.385328] LustreError: 510382:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.392834] LustreError: 510382:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [12028.897305] LustreError: 510344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195179 with bad export cookie 15692702165782597325 [12028.901755] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.903917] LustreError: 510344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12029.028664] Lustre: server umount lustre-MDT0001 complete [12032.725683] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [12033.840553] Key type lgssc unregistered [12033.984339] LNet: 518192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12033.987852] LNetError: 518192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12033.996255] LNet: Removed LNI 192.168.203.110@tcp [12034.317062] Key type .llcrypt unregistered [12034.318207] Key type ._llcrypt unregistered [12041.995609] Key type ._llcrypt registered [12041.996815] Key type .llcrypt registered [12042.032094] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [12042.440232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12042.447147] alg: No test for adler32 (adler32-zlib) [12043.295645] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [12043.375834] LNet: Added LNI 192.168.203.110@tcp [8/256/0/180] [12044.952127] Key type lgssc registered [12045.279439] Lustre: Echo OBD driver; http://www.lustre.org/ [12056.764608] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [12059.497818] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [12069.699320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12069.703566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.776434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12070.785846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12070.819992] Lustre: lustre-MDT0000: new disk, initializing [12070.839313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12070.844444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12072.078642] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [12075.937740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.977565] Lustre: 522639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12075.999919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12076.001773] Lustre: Skipped 1 previous similar message [12076.035582] Lustre: lustre-MDT0001: new disk, initializing [12076.066605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12076.078374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12076.081822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12077.271836] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [12080.692102] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12082.331627] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12084.354700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12084.446557] Lustre: lustre-OST0000: new disk, initializing [12084.448175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12084.451832] Lustre: 524605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12084.478316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12086.216147] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_default_debug -1 all [12089.703134] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12089.851391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12089.854749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12089.885159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12091.041666] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12091.109696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12092.406338] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12092.485830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12094.945977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12094.946766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12094.949389] LustreError: Skipped 1 previous similar message [12094.949607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12094.953629] Lustre: Skipped 1 previous similar message [12099.214461] Lustre: server umount lustre-OST0000 complete [12101.600724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12101.602738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12101.606912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12101.608782] Lustre: Skipped 1 previous similar message [12105.184936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12105.185377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.519177] Lustre: server umount lustre-MDT0000 complete [12107.667502] LustreError: 523432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195258 with bad export cookie 14643488588569217194 [12107.670363] LustreError: MGC192.168.203.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.671582] LustreError: 523432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12107.779591] Lustre: server umount lustre-MDT0001 complete [12115.055421] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing set_hostid [12117.987333] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing load_modules_local [12127.273891] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11803 sec ========= 09:21:18 (1786195278) [12127.817557] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:21:18 (1786195278) === [12128.835757] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:21:19 (1786195279) === [12141.274134] Lustre: DEBUG MARKER: oleg310-server.virtnet: executing unload_modules_local [12142.342654] Key type lgssc unregistered [12142.463333] LNet: 531239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12142.465582] LNetError: 531239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12142.475307] LNet: Removed LNI 192.168.203.110@tcp [12142.818316] Key type .llcrypt unregistered [12142.819418] Key type ._llcrypt unregistered