[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479728306 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003203] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.023112] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028099] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031102] *** VALIDATE tmpfs *** [ 0.032349] *** VALIDATE proc *** [ 0.033205] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.044079] debug: unmapping init [mem 0xffffffff8fa59000-0xffffffff8fa60fff] [ 0.046173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048019] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050007] ... generic registers: 4 [ 0.051008] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053011] ... fixed-purpose events: 3 [ 0.054017] ... event mask: 000000070000000f [ 0.055306] rcu: Hierarchical SRCU implementation. [ 0.057365] smp: Bringing up secondary CPUs ... [ 0.058502] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.062806] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141308] node 0 deferred pages initialised in 74ms [ 0.145083] devtmpfs: initialized [ 0.146289] x86/mm: Memory block size: 128MB [ 0.148386] gcov: version magic: 0x41383552 [ 0.153298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158307] pinctrl core: initialized pinctrl subsystem [ 0.160257] [ 0.160751] ************************************************************* [ 0.164015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ** ** [ 0.169013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171012] ** ** [ 0.174013] ** This means that this kernel is built to expose internal ** [ 0.176011] ** IOMMU data structures, which may compromise security on ** [ 0.179015] ** your system. ** [ 0.182014] ** ** [ 0.183013] ** If you see this message and you are not debugging the ** [ 0.185018] ** kernel, report this immediately to your vendor! ** [ 0.187013] ** ** [ 0.190013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193013] ************************************************************* [ 0.195782] NET: Registered protocol family 16 [ 0.197483] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207040] cpuidle: using governor menu [ 0.209547] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212555] PCI: Using configuration type 1 for base access [ 0.214133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224220] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228053] cryptd: max_cpu_qlen set to 1000 [ 0.231271] ACPI: Added _OSI(Module Device) [ 0.233017] ACPI: Added _OSI(Processor Device) [ 0.235012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241467] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248554] ACPI: Interpreter enabled [ 0.250064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251011] ACPI: Using IOAPIC for interrupt routing [ 0.253132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257435] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267226] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281365] acpiphp: Slot [2] registered [ 0.283106] acpiphp: Slot [5] registered [ 0.284108] acpiphp: Slot [6] registered [ 0.286122] acpiphp: Slot [7] registered [ 0.287092] acpiphp: Slot [8] registered [ 0.289143] acpiphp: Slot [9] registered [ 0.290106] acpiphp: Slot [10] registered [ 0.292107] acpiphp: Slot [3] registered [ 0.294125] acpiphp: Slot [4] registered [ 0.295075] acpiphp: Slot [11] registered [ 0.297092] acpiphp: Slot [12] registered [ 0.298100] acpiphp: Slot [13] registered [ 0.299086] acpiphp: Slot [14] registered [ 0.301096] acpiphp: Slot [15] registered [ 0.302085] acpiphp: Slot [16] registered [ 0.304089] acpiphp: Slot [17] registered [ 0.305162] acpiphp: Slot [18] registered [ 0.307097] acpiphp: Slot [19] registered [ 0.309121] acpiphp: Slot [20] registered [ 0.310080] acpiphp: Slot [21] registered [ 0.311078] acpiphp: Slot [22] registered [ 0.312090] acpiphp: Slot [23] registered [ 0.314084] acpiphp: Slot [24] registered [ 0.315090] acpiphp: Slot [25] registered [ 0.317105] acpiphp: Slot [26] registered [ 0.318088] acpiphp: Slot [27] registered [ 0.319091] acpiphp: Slot [28] registered [ 0.321107] acpiphp: Slot [29] registered [ 0.322091] acpiphp: Slot [30] registered [ 0.324100] acpiphp: Slot [31] registered [ 0.325041] PCI host bridge to bus 0000:00 [ 0.326024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.331020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.338022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343046] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346444] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356834] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361847] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374103] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376776] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381805] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409962] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425055] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470318] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529579] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.544017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.549014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.581208] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.588018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.615026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.627433] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.641015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.647015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.667016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.680299] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.700016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.728020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.744572] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.747000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.750816] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.755557] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.758387] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.763125] iommu: Default domain type: Passthrough [ 0.765405] SCSI subsystem initialized [ 0.767153] ACPI: bus type USB registered [ 0.768212] usbcore: registered new interface driver usbfs [ 0.771091] usbcore: registered new interface driver hub [ 0.773080] usbcore: registered new device driver usb [ 0.775175] pps_core: LinuxPPS API ver. 1 registered [ 0.776010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.780067] PTP clock support registered [ 0.782065] EDAC MC: Ver: 3.0.0 [ 0.784140] PCI: Using ACPI for IRQ routing [ 0.785623] NetLabel: Initializing [ 0.787013] NetLabel: domain hash size = 128 [ 0.788016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.790136] NetLabel: unlabeled traffic allowed by default [ 0.794121] vgaarb: loaded [ 0.795242] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.797016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.804076] clocksource: Switched to clocksource kvm-clock [ 0.927744] VFS: Disk quotas dquot_6.6.0 [ 0.929561] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.932375] *** VALIDATE ramfs *** [ 0.933812] *** VALIDATE hugetlbfs *** [ 0.935704] pnp: PnP ACPI init [ 0.938415] pnp: PnP ACPI: found 6 devices [ 0.968589] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.972612] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.975245] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.977503] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.980092] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.982788] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.986174] NET: Registered protocol family 2 [ 0.989460] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.995171] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.999444] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.004688] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.008565] TCP: Hash tables configured (established 65536 bind 65536) [ 1.011923] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.015151] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018118] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.020735] NET: Registered protocol family 1 [ 1.023561] RPC: Registered named UNIX socket transport module. [ 1.026086] RPC: Registered udp transport module. [ 1.028078] RPC: Registered tcp transport module. [ 1.030061] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.032746] NET: Registered protocol family 44 [ 1.034371] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.037879] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.040137] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.042292] PCI: CLS 0 bytes, default 64 [ 1.044699] Unpacking initramfs... [ 2.486669] debug: unmapping init [mem 0xffff95cfbcc54000-0xffff95cfbffbffff] [ 2.493028] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.495323] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.497909] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.012536] Initialise system trusted keyrings [ 3.016360] Key type blacklist registered [ 3.019061] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.029446] zbud: loaded [ 3.032448] *** VALIDATE nfs *** [ 3.033726] *** VALIDATE nfs4 *** [ 3.035196] pstore: using deflate compression [ 3.038872] Platform Keyring initialized [ 3.154600] NET: Registered protocol family 38 [ 3.156587] Key type asymmetric registered [ 3.158258] Asymmetric key parser 'x509' registered [ 3.160524] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.164161] io scheduler mq-deadline registered [ 3.166477] io scheduler kyber registered [ 3.168605] io scheduler bfq registered [ 3.170490] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.173989] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.177284] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.180583] ACPI: Power Button [PWRF] [ 3.186493] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.194702] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.207517] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.215099] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.229406] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.256567] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.285132] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.289810] Non-volatile memory driver v1.3 [ 3.291357] Linux agpgart interface v0.103 [ 3.327781] virtio_blk virtio1: [vda] 150072 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.332258] vda: detected capacity change from 0 to 76836864 [ 3.350061] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.353199] vdb: detected capacity change from 0 to 1073741824 [ 3.374986] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.378654] vdc: detected capacity change from 0 to 2621440000 [ 3.394300] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.396732] vdd: detected capacity change from 0 to 2621440000 [ 3.411665] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.414845] vde: detected capacity change from 0 to 4294967296 [ 3.432098] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.434535] vdf: detected capacity change from 0 to 4294967296 [ 3.440222] libphy: Fixed MDIO Bus: probed [ 3.445248] usbcore: registered new interface driver usbserial_generic [ 3.448060] usbserial: USB Serial support registered for generic [ 3.450662] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.455336] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.457370] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.459963] mousedev: PS/2 mouse device common for all mice [ 3.462893] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.466646] rtc_cmos 00:05: RTC can wake from S4 [ 3.472062] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.474033] rtc_cmos 00:05: registered as rtc0 [ 3.478633] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.480895] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.482796] intel_pstate: CPU model not supported [ 3.490073] hid: raw HID events driver (C) Jiri Kosina [ 3.491963] usbcore: registered new interface driver usbhid [ 3.493534] usbhid: USB HID core driver [ 3.494879] drop_monitor: Initializing network drop monitor service [ 3.496846] Initializing XFRM netlink socket [ 3.498679] NET: Registered protocol family 10 [ 3.501642] Segment Routing with IPv6 [ 3.503294] NET: Registered protocol family 17 [ 3.505945] mpls_gso: MPLS GSO support [ 3.513556] RAS: Correctable Errors collector initialized. [ 3.515482] AVX version of gcm_enc/dec engaged. [ 3.516990] AES CTR mode by8 optimization enabled [ 3.604973] sched_clock: Marking stable (3604940578, 0)->(4523891984, -918951406) [ 3.608035] registered taskstats version 1 [ 3.611196] Loading compiled-in X.509 certificates [ 3.613641] zswap: loaded using pool lzo/zbud [ 3.639142] Key type big_key registered [ 3.653333] Key type encrypted registered [ 3.655090] ima: No TPM chip found, activating TPM-bypass! [ 3.656878] ima: Allocated hash algorithm: sha1 [ 3.658901] ima: No architecture policies found [ 3.661053] evm: Initialising EVM extended attributes: [ 3.663333] evm: security.selinux [ 3.664663] evm: security.ima [ 3.665836] evm: security.capability [ 3.667365] evm: HMAC attrs: 0x1 [ 3.669978] rtc_cmos 00:05: setting system clock to 2026-09-07 01:20:48 UTC (1788744048) [ 3.676978] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 3.680543] debug: unmapping init [mem 0xffffffff8f782000-0xffffffff8fa58fff] [ 3.689181] Write protecting the kernel read-only data: 28672k [ 3.692831] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 3.695971] debug: unmapping init [mem 0xffffffff8e714000-0xffffffff8e7fffff] [ 3.738588] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.747847] systemd[1]: Detected virtualization kvm. [ 3.749806] systemd[1]: Detected architecture x86-64. [ 3.751983] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.779908] systemd[1]: No hostname configured. [ 3.782162] systemd[1]: Set hostname to . [ 3.784634] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787571] systemd[1]: Initializing machine ID from random generator. [ 3.836094] random: ln: uninitialized urandom read (6 bytes read) [ 3.944206] random: systemd: uninitialized urandom read (16 bytes read) [ 3.947205] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.955572] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.959539] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.681976] device-mapper: uevent: version 1.0.3 [ 4.684633] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook.[ 5.075095] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.494896] virtio_net virtio0 ens2: renamed from eth0 [ 5.543623] scsi host0: ata_piix [ 5.582139] scsi host1: ata_piix [ 5.583854] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.585913] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.058422] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.172845] random: crng init done [ 10.174403] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.800420] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.101490] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.375438] SELinux: Disabled at runtime. [ 12.452189] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.461299] systemd[1]: Detected virtualization kvm. [ 12.463542] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.021973] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.025982] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.031984] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.036873] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.040718] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.049819] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.059678] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ 13.184880] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.574601] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.819630] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.884751] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.064391] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.079877] EDAC sbridge: Ver: 1.1.2 [ 16.138666] Key type dns_resolver registered [ 16.461352] NFS: Registering the id_resolver key type [ 16.463215] Key type id_resolver registered [ 16.464733] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg458-server login: [ 56.479023] hrtimer: interrupt took 3151105 ns [ 59.338925] libcfs: loading out-of-tree module taints kernel. [ 59.386465] Key type ._llcrypt registered [ 59.389180] Key type .llcrypt registered [ 59.486334] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 81.068691] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 82.794512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.818919] alg: No test for adler32 (adler32-zlib) [ 84.714983] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 86.026440] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 87.975535] Key type lgssc registered [ 90.540553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.712821] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.995906] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 194.702854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.737491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.031615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.061752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.128920] Lustre: lustre-MDT0000: new disk, initializing [ 196.271706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.330802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.145650] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.006758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.122783] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.148584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.151316] Lustre: Skipped 1 previous similar message [ 216.273331] Lustre: lustre-MDT0001: new disk, initializing [ 216.423911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.467633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.480946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.521469] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.512894] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 237.736775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.021721] Lustre: lustre-OST0000: new disk, initializing [ 238.034050] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.043318] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.153919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.395813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.417409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.626427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.909510] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.770101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.063149] Lustre: lustre-OST0001: new disk, initializing [ 267.067775] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.072076] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.181552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.581973] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.018467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 274.034489] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 274.161209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 287.489172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.943750] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.947317] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.092298] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing yml_node [ 313.561439] Lustre: DEBUG MARKER: Client: 2.17.58.44 [ 316.759736] Lustre: DEBUG MARKER: MDS: 2.17.58.44 [ 319.764310] Lustre: DEBUG MARKER: OSS: 2.17.58.44 [ 321.535394] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 21:26:04 EDT 2026 [ 342.633852] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.433716] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.442651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.445352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.480463] Lustre: Skipped 3 previous similar messages [ 361.487205] Lustre: Skipped 2 previous similar messages [ 366.560403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.572359] Lustre: Skipped 4 previous similar messages [ 367.718107] Lustre: server umount lustre-MDT0000 complete [ 372.019632] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744416 with bad export cookie 16148537358653230581 [ 372.023120] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.027470] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 376.823104] LustreError: 6516:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.830239] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.833590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.847801] LustreError: 6516:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 376.875536] Lustre: Skipped 1 previous similar message [ 377.824371] LustreError: 9449:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.861423] LustreError: 9449:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 378.485196] Lustre: server umount lustre-MDT0001 complete [ 383.387935] Lustre: server umount lustre-OST0000 complete [ 387.812556] Lustre: server umount lustre-OST0001 complete [ 395.209793] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 406.372806] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 449.640930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.935930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.971284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.192461] Lustre: lustre-MDT0000: new disk, initializing [ 450.314996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.337216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 455.235364] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 465.874761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.094701] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.230328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.235481] Lustre: Skipped 1 previous similar message [ 466.356706] Lustre: lustre-MDT0001: new disk, initializing [ 466.495902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.548401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.562037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.621606] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 480.611693] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.358523] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.765056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.278191] Lustre: lustre-OST0000: new disk, initializing [ 495.284717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.302722] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.421698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.451773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.481038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.637898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.392425] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 514.288246] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.403441] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.732097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.708169] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.003556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 532.467780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.470129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.484029] LustreError: Skipped 1 previous similar message [ 532.497176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.501279] Lustre: Skipped 1 previous similar message [ 532.533227] Lustre: Skipped 4 previous similar messages [ 534.264851] Lustre: server umount lustre-OST0000 complete [ 542.689523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.700583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.705157] Lustre: Skipped 1 previous similar message [ 542.715191] Lustre: Skipped 1 previous similar message [ 544.781825] Lustre: server umount lustre-MDT0000 complete [ 547.830546] LustreError: 15089:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.856260] LustreError: 15089:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 548.546791] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744593 with bad export cookie 16148537358653232114 [ 548.550846] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.562981] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 548.940393] Lustre: server umount lustre-MDT0001 complete [ 554.375413] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:29:57 (1788744597) [ 557.590450] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 559.237459] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:30:02 (1788744602) [ 567.485289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.156082] LustreError: 19458:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.241924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.409483] LustreError: 19459:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.453642] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 582.206391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.677414] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 595.299347] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.393759] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 608.274218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.094869] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 622.872575] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 644.577053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.585562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.614056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.344577] Lustre: server umount lustre-OST0000 complete [ 654.311625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.322437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.335099] Lustre: Skipped 1 previous similar message [ 656.986559] Lustre: server umount lustre-MDT0000 complete [ 659.941800] LustreError: 19459:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.964410] LustreError: 19459:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 660.867836] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744705 with bad export cookie 16148537358653232996 [ 660.876696] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.890849] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 661.187786] Lustre: server umount lustre-MDT0001 complete [ 672.175775] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 675.329536] Key type lgssc unregistered [ 675.657466] LNet: 23020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.670739] LNetError: 23020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.693807] LNet: Removed LNI 192.168.204.158@tcp [ 676.747169] Key type .llcrypt unregistered [ 676.749064] Key type ._llcrypt unregistered [ 699.101415] Key type ._llcrypt registered [ 699.104841] Key type .llcrypt registered [ 699.215557] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 700.511950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.541815] alg: No test for adler32 (adler32-zlib) [ 701.701653] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 701.949163] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 703.623909] Key type lgssc registered [ 704.634417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 712.852302] loop: module loaded [ 712.944194] loop0: detected capacity change from 0 to 204800000 [ 713.185633] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.206663] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.682707] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.052492] loop0: detected capacity change from 0 to 204800000 [ 719.215556] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.227044] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.242858] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.814659] loop0: detected capacity change from 0 to 204800000 [ 729.059123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.085978] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.261914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 730.315234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 730.421775] Lustre: lustre-MDT0000: new disk, initializing [ 730.492699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.528071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 735.331161] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 745.230657] loop1: detected capacity change from 0 to 204800000 [ 750.958684] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 751.286447] Lustre: lustre-OST07c6: new disk, initializing [ 751.290948] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 751.296337] Lustre: Skipped 1 previous similar message [ 751.303592] Lustre: 26202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 751.381382] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 756.835943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 756.844784] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 756.867174] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 757.867801] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 762.495441] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 762.501323] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.505468] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.510281] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.514886] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 762.519492] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 780.052098] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 780.063787] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.071298] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.080492] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 780.096533] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 780.103547] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.321940] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 785.336134] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.343201] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.355662] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.364571] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.370570] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.376817] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 785.382357] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.388256] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 785.393171] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.398214] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.403455] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.615086] Lustre: 26887:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 788.622372] Lustre: 26887:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 788.633546] Lustre: 26887:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 788.639937] Lustre: 26887:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.648530] Lustre: 26887:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.653059] Lustre: 26887:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.658791] Lustre: 26887:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 788.666501] Lustre: 26887:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.676097] Lustre: 26887:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 788.689412] Lustre: 26887:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.695706] Lustre: 26887:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 788.702107] Lustre: 26887:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.633204] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 793.645508] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.652356] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.658186] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.671462] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.684279] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.693342] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.706775] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.713329] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 793.718542] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.726509] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.731444] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.789616] Lustre: 26887:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 804.810912] Lustre: 26887:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 804.818159] Lustre: 26887:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 804.830157] Lustre: 26887:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 804.841556] Lustre: 26887:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 804.849785] Lustre: 26887:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 804.855340] Lustre: 26887:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 804.870170] Lustre: 26887:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 804.876826] Lustre: 26887:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 804.889134] Lustre: 26887:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 804.904164] Lustre: 26887:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 804.913259] Lustre: 26887:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 853.986362] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 854.003080] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.031924] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 855.574698] Lustre: server umount lustre-OST07c6 complete [ 865.843877] Lustre: server umount lustre-MDT0000 complete [ 881.992547] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 884.728231] Key type lgssc unregistered [ 885.106255] LNet: 28758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.127858] LNetError: 28758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.186671] LNet: Removed LNI 192.168.204.158@tcp [ 887.079114] Key type .llcrypt unregistered [ 887.086281] Key type ._llcrypt unregistered [ 905.134893] Key type ._llcrypt registered [ 905.137147] Key type .llcrypt registered [ 905.252504] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 906.654901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.690628] alg: No test for adler32 (adler32-zlib) [ 907.885610] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 908.191988] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 909.863194] Key type lgssc registered [ 910.987809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.440676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.482018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.918834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.836425] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 929.252981] LustreError: 30041:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.268665] LustreError: 30041:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 931.661584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.979381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.092327] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 944.661133] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 950.577712] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.278905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.642977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.774133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 965.085473] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 973.738945] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 995.813349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 995.819426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.847623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.109362] Lustre: server umount lustre-OST0000 complete [ 1002.977790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.987054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.999596] Lustre: Skipped 1 previous similar message [ 1003.011588] Lustre: Skipped 1 previous similar message [ 1006.341854] Lustre: server umount lustre-MDT0000 complete [ 1008.102646] LustreError: 30042:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.120382] LustreError: 30042:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1010.175121] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745055 with bad export cookie 619097004283408253 [ 1010.178960] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.188448] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1020.641645] Lustre: server umount lustre-MDT0001 complete [ 1032.455436] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1035.334247] Key type lgssc unregistered [ 1035.690690] LNet: 33779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.698074] LNetError: 33779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.718246] LNet: Removed LNI 192.168.204.158@tcp [ 1036.644878] Key type .llcrypt unregistered [ 1036.650214] Key type ._llcrypt unregistered [ 1058.117648] Key type ._llcrypt registered [ 1058.120720] Key type .llcrypt registered [ 1058.274241] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1059.407961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.442650] alg: No test for adler32 (adler32-zlib) [ 1060.494671] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 1060.831732] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1062.521052] Key type lgssc registered [ 1063.615785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1072.682075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1072.718090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.227462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.477758] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1084.388723] LustreError: 35046:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.436383] LustreError: 35046:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1089.512885] LustreError: 35045:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.728522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.171275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1095.123588] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1103.273624] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1109.118650] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1117.566051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1117.973301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.070768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1124.724722] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1132.429434] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.830990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.841049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.873404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.924154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1149.929997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.930934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.951872] Lustre: Skipped 1 previous similar message [ 1151.685981] Lustre: server umount lustre-OST0000 complete [ 1156.577235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.584405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.605406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1160.161729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1160.162723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.172830] Lustre: server umount lustre-MDT0000 complete [ 1165.281690] LustreError: 35041:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.300702] LustreError: 35041:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1166.538964] LustreError: 35025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745211 with bad export cookie 11617610768505136743 [ 1166.549956] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.566188] LustreError: 35025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.188738] Lustre: server umount lustre-MDT0001 complete [ 1179.716582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1183.400707] Key type lgssc unregistered [ 1183.717430] LNet: 38595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.724850] LNetError: 38595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.748966] LNet: Removed LNI 192.168.204.158@tcp [ 1184.663930] Key type .llcrypt unregistered [ 1184.665707] Key type ._llcrypt unregistered [ 1211.111763] Key type ._llcrypt registered [ 1211.115340] Key type .llcrypt registered [ 1211.248339] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1212.552361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.591820] alg: No test for adler32 (adler32-zlib) [ 1213.776848] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 1214.097250] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1215.903334] Key type lgssc registered [ 1217.507803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1225.903520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1225.938657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.464435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.600729] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1237.987417] LustreError: 40009:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.013059] LustreError: 40009:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1241.532351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.970092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.550613] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1254.760657] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1261.405410] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1268.133045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.466907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.571413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1276.623654] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1284.996767] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1296.791948] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 1297.389684] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.395304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.419842] Lustre: Skipped 2 previous similar messages [ 1297.446714] Lustre: Skipped 2 previous similar messages [ 1299.734953] Lustre: server umount lustre-MDT0000 complete [ 1301.885335] LustreError: 40005:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.671733] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745348 with bad export cookie 13732531561661295672 [ 1303.681728] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.685856] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1307.003039] LustreError: 40006:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.026277] LustreError: 40006:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1307.050885] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (stopping) [ 1307.618452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.190647] Lustre: server umount lustre-MDT0001 complete [ 1325.748296] Lustre: server umount lustre-OST0000 complete [ 1341.757659] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1344.306235] Key type lgssc unregistered [ 1344.579605] LNet: 43871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.593907] LNetError: 43871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.623054] LNet: Removed LNI 192.168.204.158@tcp [ 1345.405905] Key type .llcrypt unregistered [ 1345.409873] Key type ._llcrypt unregistered [ 1370.830340] Key type ._llcrypt registered [ 1370.834102] Key type .llcrypt registered [ 1370.967661] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1372.294347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.398148] alg: No test for adler32 (adler32-zlib) [ 1373.644765] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 1373.907534] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1375.623264] Key type lgssc registered [ 1376.842043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1385.142761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1385.166667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.736223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.763278] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1396.193038] LustreError: 45286:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.221309] LustreError: 45286:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1401.122672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.312280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1401.547384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.491237] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1414.240741] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1419.972641] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1426.921125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.314147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.407659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1434.094663] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1442.208593] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.999129] Lustre: lustre-OST0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 1455.084475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.086517] Lustre: lustre-OST0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 1460.103197] Lustre: Skipped 2 previous similar messages [ 1465.213829] Lustre: lustre-OST0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 1465.224141] Lustre: Skipped 2 previous similar messages [ 1466.335276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1466.515791] Lustre: server umount lustre-OST0000 complete [ 1485.800268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.807393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.822664] Lustre: Skipped 2 previous similar messages [ 1485.827653] Lustre: Skipped 2 previous similar messages [ 1490.935977] LustreError: 45287:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.953450] LustreError: 45287:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1491.009483] Lustre: server umount lustre-MDT0000 complete [ 1494.718044] LustreError: 45266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745539 with bad export cookie 9851568181392168138 [ 1494.722871] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.741046] LustreError: 45266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1509.343952] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1509.666415] Lustre: server umount lustre-MDT0001 complete [ 1520.525812] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1523.234031] Key type lgssc unregistered [ 1523.505145] LNet: 48999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.512421] LNetError: 48999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.587741] LNet: Removed LNI 192.168.204.158@tcp [ 1525.426741] Key type .llcrypt unregistered [ 1525.429347] Key type ._llcrypt unregistered [ 1553.576371] Key type ._llcrypt registered [ 1553.583477] Key type .llcrypt registered [ 1553.687307] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1554.961082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1555.116046] alg: No test for adler32 (adler32-zlib) [ 1556.320520] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 1556.721277] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1558.415216] Key type lgssc registered [ 1559.607860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1567.659841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1567.714727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.358248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.276312] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1579.491091] LustreError: 50414:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.512208] LustreError: 50414:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1583.737682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.298211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1589.317870] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1596.721396] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1602.623442] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1610.280550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.590796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.717468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1616.207651] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1622.666293] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1636.822518] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1640.468542] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1652.284219] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0621h48m14s [ 1653.616186] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1655.779515] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.784490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.784731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1655.788021] Lustre: Skipped 2 previous similar messages [ 1655.809902] Lustre: Skipped 2 previous similar messages [ 1656.870695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.877689] Lustre: Skipped 2 previous similar messages [ 1660.312275] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 1661.960520] Lustre: server umount lustre-MDT0000 complete [ 1662.957143] LustreError: 50415:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.971687] LustreError: 50415:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1665.475625] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1667.449682] LustreError: 52514:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.675953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.827593] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1673.086835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.153133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1674.102220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1677.755398] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1681.505316] Lustre: DEBUG MARKER: Wait for df (36645) ... [ 1683.058674] Lustre: DEBUG MARKER: done [ 1688.546159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.558085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.575412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.578497] Lustre: Skipped 1 previous similar message [ 1689.571126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.694671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.704878] Lustre: Skipped 3 previous similar messages [ 1701.343128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1701.558922] Lustre: server umount lustre-OST0000 complete [ 1710.048449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.052082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1710.081819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1710.096299] Lustre: Skipped 3 previous similar messages [ 1711.480491] Lustre: server umount lustre-MDT0000 complete [ 1715.300240] LustreError: 50397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745760 with bad export cookie 11570393757965681876 [ 1715.312067] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.628437] Lustre: server umount lustre-MDT0001 complete [ 1726.938972] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1729.769366] Key type lgssc unregistered [ 1730.025702] LNet: 55356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.033143] LNetError: 55356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.050965] LNet: Removed LNI 192.168.204.158@tcp [ 1730.944570] Key type .llcrypt unregistered [ 1730.947148] Key type ._llcrypt unregistered [ 1768.217964] Key type ._llcrypt registered [ 1768.220514] Key type .llcrypt registered [ 1768.283596] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1769.143493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.176773] alg: No test for adler32 (adler32-zlib) [ 1770.182645] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 1770.391742] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1772.079285] Key type lgssc registered [ 1773.103721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1780.082637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1780.106159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.348443] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1781.367416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1781.494500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.613108] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1793.777567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.956321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1793.982971] Lustre: 57550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1794.432674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.548567] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1807.849373] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1815.869753] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1826.303265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.697320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1827.126076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.250298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1835.517295] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1844.529208] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.205576] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0621h51m34s [ 1852.732693] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1860.491650] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1863.077387] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1863.358593] LustreError: 57559:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1883.561552] Lustre: lustre-MDT0000: Client af937cdb-8566-4996-8aaa-75de7a2d1a91 (at 192.168.204.58@tcp) reconnecting [ 1903.451125] LustreError: 57559:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1906.647901] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1909.568772] Lustre: DEBUG MARKER: done [ 1920.484729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.494411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1920.505709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.509613] LustreError: Skipped 1 previous similar message [ 1920.517273] Lustre: Skipped 1 previous similar message [ 1922.369024] Lustre: server umount lustre-OST0000 complete [ 1927.648383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.654250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.670821] Lustre: Skipped 1 previous similar message [ 1927.676106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.740885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.759434] Lustre: Skipped 2 previous similar messages [ 1931.809649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.819525] Lustre: Skipped 1 previous similar message [ 1932.142746] Lustre: server umount lustre-MDT0000 complete [ 1935.976085] LustreError: 59414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745980 with bad export cookie 6281876688563392952 [ 1935.995255] LustreError: 59414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1936.002750] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1936.516776] Lustre: server umount lustre-MDT0001 complete [ 1949.982473] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1954.089580] Key type lgssc unregistered [ 1954.486942] LNet: 61883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.515930] LNetError: 61883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.552344] LNet: Removed LNI 192.168.204.158@tcp [ 1955.696641] Key type .llcrypt unregistered [ 1955.698946] Key type ._llcrypt unregistered [ 2002.018750] Key type ._llcrypt registered [ 2002.021348] Key type .llcrypt registered [ 2002.115275] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2003.778183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.827821] alg: No test for adler32 (adler32-zlib) [ 2004.880899] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 2005.150731] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2006.863342] Key type lgssc registered [ 2008.320612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2012.105579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2012.125200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.359564] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2013.378235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.563524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.447938] Lustre: Failing over lustre-MDT0000 [ 2014.842821] Lustre: server umount lustre-MDT0000 complete [ 2022.667646] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:54:25 (1788746065) [ 2030.226346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.607117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.815227] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2043.502026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.585951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2043.628744] Lustre: 64679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2043.947992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.293542] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2057.501049] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.479944] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.458361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.607248] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2071.787927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.427220] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2079.431891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2088.149082] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2093.797566] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2105.315802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.319830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.331258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2105.359233] Lustre: Skipped 2 previous similar messages [ 2105.397696] Lustre: Skipped 2 previous similar messages [ 2106.562330] Lustre: server umount lustre-MDT0000 complete [ 2110.112900] LustreError: 65413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746154 with bad export cookie 16732920400767002882 [ 2110.116485] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.126039] LustreError: 65413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2110.433350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.531305] Lustre: server umount lustre-MDT0001 complete [ 2114.157920] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2118.602559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2126.291272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.363296] LustreError: 66527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.386567] LustreError: 66527:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2135.480467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.551341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2136.490339] LustreError: 66528:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.494046] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2139.767354] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2141.681737] LustreError: 66527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.794975] LustreError: 66528:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.764349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.162909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.215227] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2154.583356] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2162.785378] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2168.549477] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2175.968859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2175.977598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.998435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.000550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.527250] Lustre: server umount lustre-MDT0000 complete [ 2182.113097] LustreError: 66527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.135559] LustreError: 66527:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2185.036332] LustreError: 64669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746229 with bad export cookie 16732920400767005157 [ 2185.054022] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.286429] Lustre: server umount lustre-MDT0001 complete [ 2191.677467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2201.136505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.615487] Lustre: 63601:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746232/real 1788746232] req@00000000455ed791 x1875636177145472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2203.635149] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.642970] Lustre: Skipped 2 previous similar messages [ 2206.751569] Lustre: 63601:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746235/real 1788746235] req@00000000473bde51 x1875636177145600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2208.736311] Lustre: 63600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746237/real 1788746237] req@000000007031e9ce x1875636177145728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746253 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.137215] LustreError: 66528:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.232636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.288549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2211.232249] Lustre: 63599:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746239/real 1788746239] req@00000000333f15a4 x1875636177145984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2215.267442] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2215.330357] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746244/real 1788746244] req@000000006ea06ec8 x1875636177146112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2224.618595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.267231] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2230.603404] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2232.365874] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2239.779290] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.370706] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2255.843186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2255.860073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.885698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.895744] Lustre: Skipped 2 previous similar messages [ 2263.019323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.027045] Lustre: Skipped 5 previous similar messages [ 2269.151188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2269.334249] Lustre: server umount lustre-MDT0000 complete [ 2273.135918] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746317 with bad export cookie 16732920400767006536 [ 2273.142863] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.149709] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2273.248147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.249783] LustreError: 71133:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.266921] Lustre: Skipped 2 previous similar messages [ 2273.272806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2273.294759] LustreError: 71133:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 2273.320696] Lustre: Skipped 3 previous similar messages [ 2273.492907] Lustre: server umount lustre-MDT0001 complete [ 2279.632981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2283.129852] Lustre: DEBUG MARKER: files compared the same [ 2285.687271] Lustre: server umount lustre-OST0000 complete [ 2301.584401] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2304.263112] Key type lgssc unregistered [ 2304.525855] LNet: 74406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.532896] LNetError: 74406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.554226] LNet: Removed LNI 192.168.204.158@tcp [ 2305.247194] Key type .llcrypt unregistered [ 2305.250899] Key type ._llcrypt unregistered [ 2329.137416] Key type ._llcrypt registered [ 2329.140809] Key type .llcrypt registered [ 2329.259982] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2330.142000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.151597] alg: No test for adler32 (adler32-zlib) [ 2331.577211] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 2331.873504] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2333.543384] Key type lgssc registered [ 2335.003284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2343.956325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.991200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.430086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.651409] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug +malloc all [ 2355.701122] LustreError: 75818:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.725538] LustreError: 75818:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2360.804789] LustreError: 75819:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.159147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.524119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2367.440846] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug +malloc all [ 2375.480932] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.876679] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.647782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.024498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.303684] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug +malloc all [ 2408.012104] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2415.597427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2430.949244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.953937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.974340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.990803] Lustre: Skipped 1 previous similar message [ 2433.674578] Lustre: server umount lustre-OST0000 complete [ 2438.624821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2438.640153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.656883] Lustre: Skipped 1 previous similar message [ 2438.661782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.186671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.186815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.214111] Lustre: Skipped 1 previous similar message [ 2444.087903] Lustre: server umount lustre-MDT0000 complete [ 2446.304974] LustreError: 75813:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.322558] LustreError: 75813:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2448.228959] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746493 with bad export cookie 10367526392691912530 [ 2448.230611] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.252558] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2448.583451] Lustre: server umount lustre-MDT0001 complete [ 2462.349827] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2466.095930] Key type lgssc unregistered [ 2466.623940] LNet: 79387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.660972] LNetError: 79387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.702463] LNet: Removed LNI 192.168.204.158@tcp [ 2467.730627] Key type .llcrypt unregistered [ 2467.732536] Key type ._llcrypt unregistered [ 2504.235309] Key type ._llcrypt registered [ 2504.238516] Key type .llcrypt registered [ 2504.422537] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2506.113718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2506.148180] alg: No test for adler32 (adler32-zlib) [ 2507.210650] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 2507.543407] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2509.362128] Key type lgssc registered [ 2510.519346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2518.560041] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2518.603513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.304271] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2539.423382] LustreError: 80777:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.158@tcp: failed processing log, type 4: rc = -110 [ 2566.177062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.368888] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2580.836558] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2588.854480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.948392] Lustre: *** cfs_fail_loc=706, val=0*** [ 2589.420178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.494333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2594.002163] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2594.785319] LustreError: 80802:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.812092] LustreError: 80802:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2599.918080] LustreError: 80803:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.956289] LustreError: 80802:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.975411] LustreError: 80802:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2602.945313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.352013] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2607.494107] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2615.782711] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2622.122842] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2625.505882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2625.514769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.534078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.092031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.096393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.112576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.046893] Lustre: server umount lustre-OST0000 complete [ 2639.334047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.335142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.347648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.354908] Lustre: Skipped 1 previous similar message [ 2641.540677] Lustre: server umount lustre-MDT0000 complete [ 2644.450966] LustreError: 81862:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.471612] LustreError: 81862:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2645.168043] LustreError: 83640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746689 with bad export cookie 17515016197407473959 [ 2645.186450] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.188043] LustreError: 83640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2645.623846] Lustre: server umount lustre-MDT0001 complete [ 2657.331315] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2659.981167] Key type lgssc unregistered [ 2660.276901] LNet: 84390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.287413] LNetError: 84390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2660.303942] LNet: Removed LNI 192.168.204.158@tcp [ 2661.370707] Key type .llcrypt unregistered [ 2661.377236] Key type ._llcrypt unregistered [ 2687.896448] Key type ._llcrypt registered [ 2687.901252] Key type .llcrypt registered [ 2688.015653] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2689.183093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.338725] alg: No test for adler32 (adler32-zlib) [ 2690.516913] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 2690.807492] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2692.463530] Key type lgssc registered [ 2693.682775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2701.682613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2701.729699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.616478] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2714.787159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.257207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2719.815947] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2720.742617] LustreError: 86491:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.773762] LustreError: 86491:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2724.839198] LustreError: 86494:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.857250] LustreError: 86494:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2728.518587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.801522] LustreError: 86490:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.828246] LustreError: 86490:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2728.847942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.050342] LustreError: 86494:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.072471] LustreError: 86494:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2735.218562] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2739.178748] LustreError: 86495:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.203495] LustreError: 86495:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2743.852375] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.872029] Lustre: MGS: Client b392b489-b959-4520-abf8-1a3db978ec0c (at 0@lo) reconnecting [ 2743.906449] LustreError: 86505:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2744.210430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.267279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2748.684510] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2764.771448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.801695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.813824] Lustre: Skipped 1 previous similar message [ 2768.953434] Lustre: server umount lustre-OST0000 complete [ 2775.010685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.036363] Lustre: Skipped 1 previous similar message [ 2775.045457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.052175] Lustre: Skipped 1 previous similar message [ 2779.583374] Lustre: server umount lustre-MDT0000 complete [ 2780.131096] LustreError: 86495:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.166473] LustreError: 86495:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2784.778808] Lustre: server umount lustre-MDT0001 complete [ 2789.512334] Lustre: server umount MGS complete [ 2800.722589] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:07:23 (1788746843) [ 2829.253902] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 2839.927828] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2875.848291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.119517] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2889.929647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.162980] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2890.220322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2890.313731] Lustre: lustre-MDT0001: new disk, initializing [ 2890.397426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.421949] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2890.431687] Lustre: 93659:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2890.450821] LustreError: 93659:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2894.808331] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2903.897990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.102479] Lustre: lustre-OST0000: new disk, initializing [ 2904.106865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2904.113599] Lustre: 94424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2904.180182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.269072] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2912.355672] Lustre: 94838:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000060f23ef1 x1875636896440576/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788746973 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2920.216445] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2927.946217] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.956726] Lustre: MGS: Client cb40011c-e764-428f-ba2f-03a046e513a4 (at 0@lo) reconnecting [ 2927.971446] LustreError: 93645:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2928.004709] Lustre: 92926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.051154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2928.141533] Lustre: lustre-MDT0000: new disk, initializing [ 2928.304433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.325282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2928.341323] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2929.380262] Lustre: 95492:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005c0a9b6f x1875636896447616/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788746990 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2933.361493] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2934.497814] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.495362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2938.515435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2938.580518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2964.459482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2964.472439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.496359] Lustre: Skipped 1 previous similar message [ 2964.503193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2965.472551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2970.524329] Lustre: server umount lustre-OST0000 complete [ 2976.235090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2976.236520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.249220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.254428] Lustre: Skipped 2 previous similar messages [ 2976.265165] Lustre: Skipped 3 previous similar messages [ 2980.623613] Lustre: server umount lustre-MDT0000 complete [ 2981.346130] LustreError: 93634:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.359835] LustreError: 93634:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2990.587863] Lustre: server umount lustre-MDT0001 complete [ 2993.778519] Lustre: server umount MGS complete [ 3005.034378] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:10:47 (1788747047) [ 3021.986130] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3024.385691] Key type lgssc unregistered [ 3024.676179] LNet: 98003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3024.684509] LNetError: 98003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3024.702405] LNet: Removed LNI 192.168.204.158@tcp [ 3025.472456] Key type .llcrypt unregistered [ 3025.477416] Key type ._llcrypt unregistered [ 3043.521428] Key type ._llcrypt registered [ 3043.523161] Key type .llcrypt registered [ 3043.651716] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3044.843275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3044.929147] alg: No test for adler32 (adler32-zlib) [ 3046.094723] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 3046.397979] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3048.111818] Key type lgssc registered [ 3049.335054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3057.613224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3057.627397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.109675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3063.636864] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3068.414485] LustreError: 99222:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.457414] LustreError: 99222:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3073.403169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.508083] LustreError: 99222:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.775556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3078.615131] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3087.320546] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.914078] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3101.553538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.056574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.147551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3109.169941] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3116.863884] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3128.800476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3128.805136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.817953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3128.824286] Lustre: Skipped 1 previous similar message [ 3133.935348] LustreError: 101045:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.952766] LustreError: 101045:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3134.115296] Lustre: server umount lustre-OST0000 complete [ 3139.041773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.054047] Lustre: Skipped 2 previous similar messages [ 3139.054645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.063361] Lustre: Skipped 1 previous similar message [ 3143.528981] Lustre: server umount lustre-MDT0000 complete [ 3144.162481] LustreError: 101415:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.192421] LustreError: 101415:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3147.086177] LustreError: 99203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747191 with bad export cookie 15877285228236248438 [ 3147.091136] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.101195] LustreError: 99203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.355547] Lustre: server umount lustre-MDT0001 complete [ 3159.161521] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3161.913390] Key type lgssc unregistered [ 3162.333954] LNet: 102755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.344471] LNetError: 102755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.376935] LNet: Removed LNI 192.168.204.158@tcp [ 3164.258081] Key type .llcrypt unregistered [ 3164.260832] Key type ._llcrypt unregistered [ 3185.717929] Key type ._llcrypt registered [ 3185.720139] Key type .llcrypt registered [ 3185.816893] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3187.249702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.283729] alg: No test for adler32 (adler32-zlib) [ 3188.602341] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 3188.969913] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3190.719981] Key type lgssc registered [ 3192.986426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3201.161979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3201.166992] LustreError: 103837:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3201.170533] Lustre: Skipped 1 previous similar message [ 3201.170569] LustreError: 103838:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3201.210442] LustreError: 103837:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3201.213935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.222772] LustreError: 103837:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3201.222801] LustreError: 103837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3203.069093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.977527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.160194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3216.582760] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3221.686504] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing lsmod [ 3224.471210] LustreError: 105033:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3224.506392] LustreError: 105043:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3224.511310] LustreError: 105033:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3224.515081] LustreError: 105033:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3224.525643] LustreError: 105033:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3224.528137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.829390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.181259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.191119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.206142] Lustre: Skipped 1 previous similar message [ 3234.036148] Lustre: server umount lustre-MDT0000 complete [ 3235.303670] LustreError: 103876:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.326699] LustreError: 103876:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 3236.323904] LustreError: 103870:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.340802] LustreError: 103870:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3237.897570] LustreError: 103857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747282 with bad export cookie 11098113200031626989 [ 3237.899864] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.906492] LustreError: 103857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3238.375748] LustreError: 104306:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f3d73750 x1875637418610816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3238.501227] Lustre: server umount lustre-MDT0001 complete [ 3252.453844] Lustre: server umount lustre-OST0000 complete [ 3260.601395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.150956] LustreError: 106109:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.244161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.069127] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3266.528607] LustreError: 106110:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.651091] LustreError: 106109:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.318907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.843439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.984321] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3289.216783] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3296.205953] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3303.232289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.596341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.677364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3309.679285] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3318.176083] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3331.557667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.569518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.574301] Lustre: Skipped 1 previous similar message [ 3331.575509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3331.575513] Lustre: Skipped 2 previous similar messages [ 3332.581394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3332.594402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.606160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.678319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.692673] Lustre: Skipped 1 previous similar message [ 3337.310699] Lustre: server umount lustre-OST0000 complete [ 3341.795356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.804312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3341.813869] Lustre: Skipped 1 previous similar message [ 3347.335601] Lustre: server umount lustre-MDT0000 complete [ 3350.991525] LustreError: 106089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747395 with bad export cookie 11098113200031627703 [ 3350.997240] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.012406] LustreError: 106089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3351.278625] Lustre: server umount lustre-MDT0001 complete [ 3362.074287] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3365.127509] Key type lgssc unregistered [ 3365.531346] LNet: 109657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3365.545681] LNetError: 109657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3365.566426] LNet: Removed LNI 192.168.204.158@tcp [ 3366.375283] Key type .llcrypt unregistered [ 3366.378439] Key type ._llcrypt unregistered [ 3393.287835] Key type ._llcrypt registered [ 3393.290784] Key type .llcrypt registered [ 3393.439166] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3394.517665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3394.652651] alg: No test for adler32 (adler32-zlib) [ 3396.096799] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 3396.433597] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3398.191205] Key type lgssc registered [ 3399.571782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3408.080460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3408.100317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.382320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.837801] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3419.616968] LustreError: 111071:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.642730] LustreError: 111071:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3422.789473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.190163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.209922] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3434.862896] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3440.468318] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3446.460069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3446.738292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.762664] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3454.977795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3460.668384] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3465.669695] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3469.822286] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3474.147425] Lustre: server umount lustre-OST0000 complete [ 3479.525150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.540167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.555340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.550059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.553388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.595888] Lustre: Skipped 1 previous similar message [ 3485.665663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.678022] Lustre: Skipped 1 previous similar message [ 3487.851304] Lustre: server umount lustre-MDT0000 complete [ 3490.788161] LustreError: 111071:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.817091] LustreError: 111071:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3491.721476] LustreError: 111051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747536 with bad export cookie 11421923999779130990 [ 3491.734306] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3502.188833] Lustre: server umount lustre-MDT0001 complete [ 3513.414590] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3516.128573] Key type lgssc unregistered [ 3516.352553] LNet: 114707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3516.358917] LNetError: 114707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.416880] LNet: Removed LNI 192.168.204.158@tcp [ 3518.253414] Key type .llcrypt unregistered [ 3518.256652] Key type ._llcrypt unregistered [ 3537.009380] Key type ._llcrypt registered [ 3537.011411] Key type .llcrypt registered [ 3537.111995] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3537.954159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3537.983544] alg: No test for adler32 (adler32-zlib) [ 3539.200586] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 3539.492245] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3541.175337] Key type lgssc registered [ 3542.397680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3551.140789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3551.162066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.542590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.738954] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3563.005639] LustreError: 115930:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.029443] LustreError: 115930:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3565.924809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.459548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3571.559360] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3579.180710] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3585.401384] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3592.423256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.825779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.938792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3600.440303] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3608.757272] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3621.857854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.861463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3621.880772] Lustre: Skipped 1 previous similar message [ 3626.986394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.990536] Lustre: Skipped 2 previous similar messages [ 3627.596375] Lustre: server umount lustre-OST0000 complete [ 3632.097314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.100927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.119460] Lustre: Skipped 1 previous similar message [ 3632.135880] Lustre: Skipped 1 previous similar message [ 3637.217866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3637.236813] Lustre: Skipped 1 previous similar message [ 3638.282676] Lustre: server umount lustre-MDT0000 complete [ 3642.336437] LustreError: 116922:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.359516] LustreError: 116922:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3642.818419] LustreError: 115911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747687 with bad export cookie 14711027742230248461 [ 3642.837613] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3643.309292] Lustre: server umount lustre-MDT0001 complete [ 3658.811880] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3662.367183] Key type lgssc unregistered [ 3662.693616] LNet: 119486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3662.702197] LNetError: 119486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3662.726032] LNet: Removed LNI 192.168.204.158@tcp [ 3663.813688] Key type .llcrypt unregistered [ 3663.819226] Key type ._llcrypt unregistered [ 3692.489542] Key type ._llcrypt registered [ 3692.492238] Key type .llcrypt registered [ 3692.618580] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3694.249798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3694.335180] alg: No test for adler32 (adler32-zlib) [ 3695.431318] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 3695.771315] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3697.495219] Key type lgssc registered [ 3699.014889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3708.257155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3708.274980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.994880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.368303] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3720.166318] LustreError: 120900:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.195396] LustreError: 120900:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3724.943208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.310982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3730.943810] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3741.364405] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.632222] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3756.945194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3757.194305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.647674] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3772.110397] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3779.779579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3788.629569] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3792.691913] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3792.700213] Lustre: Skipped 2 previous similar messages [ 3797.075746] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3797.108115] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3797.580483] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3797.585672] Lustre: Skipped 2 previous similar messages [ 3803.705313] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3803.710595] Lustre: Skipped 1 previous similar message [ 3809.624884] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.58@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3809.637338] Lustre: Skipped 2 previous similar messages [ 3813.552258] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.58@tcp 192.168.0.[2,10]@tcp [ 3823.002413] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.58/24@tcp in log lustre-MDT0000 [ 3823.009903] Lustre: Skipped 2 previous similar messages [ 3823.249651] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.58/24@tcp [ 3823.268111] Lustre: Skipped 1 previous similar message [ 3833.836359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3833.841958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.852739] LustreError: Skipped 3 previous similar messages [ 3833.875748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.388244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.388367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.407480] Lustre: Skipped 1 previous similar message [ 3838.557839] Lustre: server umount lustre-OST0000 complete [ 3843.041441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3843.050729] LustreError: Skipped 2 previous similar messages [ 3843.060343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.074512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.626343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.640456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.658375] Lustre: Skipped 1 previous similar message [ 3848.976265] Lustre: server umount lustre-MDT0000 complete [ 3851.745594] LustreError: 120900:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.770783] LustreError: 120900:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3852.548810] LustreError: 120881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747897 with bad export cookie 1557365841951790290 [ 3852.555231] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.560709] LustreError: 120881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3852.814719] Lustre: server umount lustre-MDT0001 complete [ 3864.996884] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3867.866721] Key type lgssc unregistered [ 3868.189739] LNet: 125720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3868.202699] LNetError: 125720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3868.233244] LNet: Removed LNI 192.168.204.158@tcp [ 3869.174863] Key type .llcrypt unregistered [ 3869.179345] Key type ._llcrypt unregistered [ 3902.228772] Key type ._llcrypt registered [ 3902.231337] Key type .llcrypt registered [ 3902.330713] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3903.468420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3903.497588] alg: No test for adler32 (adler32-zlib) [ 3904.581641] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 3904.868892] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3906.583419] Key type lgssc registered [ 3907.981947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3916.429848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3916.475758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.968112] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3917.996597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.004254] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.58/24@tcp [ 3922.984692] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3927.532026] LustreError: 127287:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.581171] LustreError: 127287:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3932.650537] LustreError: 127288:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.970234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.784607] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3934.826024] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.58/24@tcp [ 3934.957383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.774245] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3951.125417] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.984776] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3966.030863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.333155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.286979] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3979.963084] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.861441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4005.346226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4005.371216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.411692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.430911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.439263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.439448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.466950] Lustre: Skipped 1 previous similar message [ 4010.369974] Lustre: server umount lustre-OST0000 complete [ 4016.608893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4016.617804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.641482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.659561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.663735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.126350] Lustre: server umount lustre-MDT0000 complete [ 4023.778598] LustreError: 127287:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.792276] LustreError: 127287:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4025.072936] LustreError: 127269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748069 with bad export cookie 6181242850662675718 [ 4025.079572] LustreError: 127269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4025.084540] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4025.477732] Lustre: server umount lustre-MDT0001 complete [ 4037.854168] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4040.605389] Key type lgssc unregistered [ 4040.895814] LNet: 130903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4040.902846] LNetError: 130903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4040.921243] LNet: Removed LNI 192.168.204.158@tcp [ 4041.816463] Key type .llcrypt unregistered [ 4041.818492] Key type ._llcrypt unregistered [ 4068.388883] Key type ._llcrypt registered [ 4068.391700] Key type .llcrypt registered [ 4068.475344] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4083.515687] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4085.059287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4085.290390] alg: No test for adler32 (adler32-zlib) [ 4086.570467] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 4086.875183] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4088.591344] Key type lgssc registered [ 4089.925784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4131.261662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4131.332043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.605223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4132.677779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4132.829223] Lustre: lustre-MDT0000: new disk, initializing [ 4132.960811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.990301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4139.658601] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4151.498703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.658776] Lustre: 134308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4151.719661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4151.723331] Lustre: Skipped 1 previous similar message [ 4151.872585] Lustre: lustre-MDT0001: new disk, initializing [ 4152.052416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4152.132981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4152.150686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4157.168567] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4169.601367] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4177.211167] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4185.334612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.755722] Lustre: lustre-OST0000: new disk, initializing [ 4185.764862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4185.781472] Lustre: 136281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4185.951254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.395234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4193.407630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4193.615269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4194.095209] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4205.189036] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4209.695659] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4209.914233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4214.908916] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4215.134703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4217.330917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.333700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4217.353918] Lustre: Skipped 1 previous similar message [ 4222.439798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4222.446165] Lustre: Skipped 1 previous similar message [ 4223.387649] Lustre: server umount lustre-OST0000 complete [ 4227.556229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.576449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4227.583222] Lustre: Skipped 2 previous similar messages [ 4232.679214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.689162] Lustre: Skipped 1 previous similar message [ 4237.801613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.808637] Lustre: Skipped 1 previous similar message [ 4241.375176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4241.704168] Lustre: server umount lustre-MDT0000 complete [ 4245.376072] LustreError: 134301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748290 with bad export cookie 12062660882348401833 [ 4245.379783] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.385539] LustreError: 134301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4245.887711] Lustre: server umount lustre-MDT0001 complete [ 4254.216215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.994299] LustreError: 138497:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.020360] LustreError: 138497:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4255.106345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.210346] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4260.322985] LustreError: 138497:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.343435] LustreError: 138497:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4264.416697] LustreError: 138498:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.845462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.337622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4273.156830] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4280.194262] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4285.952646] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.861690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.133875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.436616] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4308.216864] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4313.139258] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4313.346445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4318.173471] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4318.486772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4327.534996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.782519] Lustre: lustre-OST0001: Not available for connect from 192.168.204.58@tcp (not set up) [ 4327.796470] Lustre: lustre-OST0001: new disk, initializing [ 4327.805811] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4327.814725] Lustre: 141872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4327.924578] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4334.903608] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4337.753193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4337.764474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4337.817921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4343.715762] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4343.916919] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4348.368848] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4348.609051] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4355.141697] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc9d05d3800.ost_server_uuid 50 [ 4356.815188] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc9d05d3800.ost_server_uuid in IDLE state after 0 sec [ 4367.460786] Lustre: server umount lustre-OST0001 complete [ 4368.865252] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4368.878296] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.886171] LustreError: 140321:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.900929] Lustre: Skipped 2 previous similar messages [ 4368.929374] LustreError: 140321:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4371.941208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4371.946060] LustreError: Skipped 1 previous similar message [ 4371.948320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.954864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.992690] LustreError: 140322:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.012287] LustreError: 140322:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4377.516351] Lustre: server umount lustre-OST0000 complete [ 4382.181050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4382.196263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.209747] Lustre: Skipped 1 previous similar message [ 4387.687615] Lustre: server umount lustre-MDT0000 complete [ 4389.346376] LustreError: 138497:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.355978] LustreError: 138497:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4391.385669] LustreError: 139234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748436 with bad export cookie 12062660882348402715 [ 4391.388730] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.410194] LustreError: 139234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4391.725328] Lustre: server umount lustre-MDT0001 complete [ 4415.141695] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4418.475671] Key type lgssc unregistered [ 4418.934263] LNet: 144870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.951898] LNetError: 144870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4420.013564] LNet: Removed LNI 192.168.204.158@tcp [ 4421.021992] Key type .llcrypt unregistered [ 4421.023718] Key type ._llcrypt unregistered [ 4463.282467] Key type ._llcrypt registered [ 4463.284188] Key type .llcrypt registered [ 4463.434381] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4477.672992] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4479.001356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4479.031040] alg: No test for adler32 (adler32-zlib) [ 4480.076252] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 4480.346360] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4482.183700] Key type lgssc registered [ 4483.252326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4523.241981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4523.265767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.722953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4524.765155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4524.858229] Lustre: lustre-MDT0000: new disk, initializing [ 4524.916279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.936571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4530.025336] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4542.737421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.919726] Lustre: 149023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4543.008883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4543.012697] Lustre: Skipped 1 previous similar message [ 4543.096367] Lustre: lustre-MDT0001: new disk, initializing [ 4543.221276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4543.295881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4543.308859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4549.062411] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4559.394875] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4565.677826] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4573.172679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.470379] Lustre: lustre-OST0000: new disk, initializing [ 4573.477958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4573.497093] Lustre: 150993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4573.603045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4580.797246] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4582.479775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4582.490313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4582.598467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4590.421827] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4595.241884] Lustre: Failing over lustre-OST0000 [ 4595.448564] Lustre: server umount lustre-OST0000 complete [ 4613.600166] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788748642/real 1788748642] req@0000000070709464 x1875638773385088/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788748658 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4613.634844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.826302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.226433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.265691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.289200] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4614.551580] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4614.631210] Lustre: 147246:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788748643/real 1788748643] req@000000008c4e191b x1875638773386112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788748659 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4619.295262] Lustre: 147248:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788748648/real 1788748648] req@000000007763371a x1875638773387392/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788748664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4619.324687] Lustre: 147248:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4620.170516] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4623.382584] Lustre: Failing over lustre-MDT0000 [ 4623.397321] LustreError: 147244:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4623.404541] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4623.412800] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4624.415533] Lustre: 147248:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788748653/real 1788748653] req@0000000025938051 x1875638773388416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788748669 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4624.435522] Lustre: 147248:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4625.376952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.382628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.620283] Lustre: server umount lustre-MDT0000 complete [ 4628.466783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.488968] Lustre: Skipped 1 previous similar message [ 4629.599165] Lustre: 147246:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788748658/real 1788748658] req@00000000e055a166 x1875638773389568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788748674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4629.879909] LustreError: 149030:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4629.893355] LustreError: 149030:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 4633.578046] LustreError: 149034:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.595228] LustreError: 149034:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4634.996824] LustreError: 149029:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.689340] LustreError: 149031:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.712662] LustreError: 149031:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4643.821572] LustreError: 149029:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.843428] LustreError: 149029:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 4644.200049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.401628] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4644.605290] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4644.671657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.741363] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4646.259839] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4648.997576] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4649.982216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4650.002673] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4650.074400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4659.184316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4659.191836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.214984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.210881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.226935] Lustre: Skipped 1 previous similar message [ 4664.147991] Lustre: server umount lustre-OST0000 complete [ 4670.435360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.436965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4670.444318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4670.455131] Lustre: Skipped 2 previous similar messages [ 4675.197789] Lustre: server umount lustre-MDT0000 complete [ 4675.561471] LustreError: 149031:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.598562] LustreError: 149031:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4678.930796] LustreError: 149828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748723 with bad export cookie 12068448716445294386 [ 4678.932123] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4678.941618] LustreError: 149828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4679.307393] Lustre: server umount lustre-MDT0001 complete [ 4690.733747] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4693.461427] Key type lgssc unregistered [ 4693.751888] LNet: 154650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4693.772564] LNetError: 154650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4693.800622] LNet: Removed LNI 192.168.204.158@tcp [ 4694.731626] Key type .llcrypt unregistered [ 4694.734447] Key type ._llcrypt unregistered [ 4730.888461] Key type ._llcrypt registered [ 4730.890989] Key type .llcrypt registered [ 4730.997254] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4732.033777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.044114] alg: No test for adler32 (adler32-zlib) [ 4733.078548] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 4733.417482] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4735.127258] Key type lgssc registered [ 4736.239396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4743.114080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4743.160559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.712881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.119883] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4754.915116] LustreError: 156181:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4754.938368] LustreError: 156181:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4758.856217] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4760.033882] LustreError: 156180:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.541383] LustreError: 156209:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4762.554288] LustreError: 156209:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4762.567274] LustreError: 156209:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4762.874890] Lustre: server umount lustre-MDT0000 complete [ 4775.949236] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4784.961499] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4816.634553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.835497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.854928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4816.907865] Lustre: lustre-MDT0000: new disk, initializing [ 4816.995515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.011788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4820.393378] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4829.534982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.647935] Lustre: 159513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4829.700702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4829.704361] Lustre: Skipped 1 previous similar message [ 4829.789343] Lustre: lustre-MDT0001: new disk, initializing [ 4829.890070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4829.933850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4829.943190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4833.614801] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4841.796887] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4846.531295] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4851.913356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.187338] Lustre: lustre-OST0000: new disk, initializing [ 4852.191860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4852.198554] Lustre: 161479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4852.291781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.693976] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4862.008789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4862.019208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4862.192511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4867.070590] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4871.654051] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4873.055309] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:41:56 (1788748916) [ 5122.016655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5122.022495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.038740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.552132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.555819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.446193] Lustre: server umount lustre-MDT0000 complete [ 5128.674914] LustreError: 160348:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.684075] LustreError: 160348:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5130.608378] LustreError: 160306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749175 with bad export cookie 12503646828870786317 [ 5130.614367] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.616874] LustreError: 160306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5133.793834] LustreError: 159519:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.796424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.834315] Lustre: Skipped 1 previous similar message [ 5133.842554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5133.852454] Lustre: Skipped 2 previous similar messages [ 5136.920612] Lustre: server umount lustre-MDT0001 complete [ 5154.271395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5154.448881] Lustre: server umount lustre-OST0000 complete [ 5162.536850] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 5171.089053] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5200.835510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.012508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5201.047615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5201.116425] Lustre: lustre-MDT0000: new disk, initializing [ 5201.176888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.189701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5205.211671] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5213.523550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.632700] Lustre: 166136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5213.687785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5213.692178] Lustre: Skipped 1 previous similar message [ 5213.766307] Lustre: lustre-MDT0001: new disk, initializing [ 5213.851298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5213.892768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5213.901738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5217.830700] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5226.558938] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5231.537971] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5236.693810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.894971] Lustre: lustre-OST0000: new disk, initializing [ 5236.898206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5236.902913] Lustre: 168107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5236.961326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.736713] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5242.433322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5242.444274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5242.572509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5249.361854] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5252.576521] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5252.728791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5255.931430] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5256.060555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5258.208848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5258.211828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5258.214497] LustreError: Skipped 1 previous similar message [ 5258.215712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5258.222318] Lustre: Skipped 1 previous similar message [ 5258.231160] Lustre: Skipped 2 previous similar messages [ 5263.333930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5263.338322] Lustre: Skipped 1 previous similar message [ 5263.562402] Lustre: server umount lustre-OST0000 complete [ 5268.450153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.438713] Lustre: server umount lustre-MDT0000 complete [ 5273.569812] LustreError: 167091:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.586766] LustreError: 167091:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5275.637767] LustreError: 169647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749320 with bad export cookie 12503646828870945147 [ 5275.639679] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.645322] LustreError: 169647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5275.962870] Lustre: server umount lustre-MDT0001 complete [ 5282.624784] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:48:45 (1788749325) [ 5294.513507] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 5302.631582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5330.705030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5330.915350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5330.941700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5331.004986] Lustre: lustre-MDT0000: new disk, initializing [ 5331.095557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.116827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5334.247253] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5342.080220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.180277] Lustre: 172859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5342.399873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5346.257163] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5354.346655] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5358.987515] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5364.796475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5365.082437] Lustre: lustre-OST0000: new disk, initializing [ 5365.085205] Lustre: Skipped 1 previous similar message [ 5365.089500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5365.092943] Lustre: Skipped 2 previous similar messages [ 5365.100157] Lustre: 174838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5370.483450] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5374.496880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5374.503859] Lustre: Skipped 1 previous similar message [ 5374.509455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5374.564661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5378.249632] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5384.677408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.681677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5384.682881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5384.682888] Lustre: Skipped 2 previous similar messages [ 5384.686635] Lustre: Skipped 2 previous similar messages [ 5384.705409] LustreError: Skipped 1 previous similar message [ 5389.511547] Lustre: server umount lustre-OST0000 complete [ 5393.889160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5398.382286] Lustre: server umount lustre-MDT0000 complete [ 5400.034122] LustreError: 172867:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.046236] LustreError: 172867:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5401.009259] LustreError: 172852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749445 with bad export cookie 12503646828870946029 [ 5401.014254] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5401.019189] LustreError: 172852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5401.247480] Lustre: server umount lustre-MDT0001 complete [ 5406.942493] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:50:50 (1788749450) [ 5418.199559] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 5424.992908] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5451.422844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.609474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5451.640845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5451.737820] Lustre: lustre-MDT0000: new disk, initializing [ 5451.795438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.801112] Lustre: Skipped 2 previous similar messages [ 5451.816301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5454.962077] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5462.372294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.471446] Lustre: 179158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5462.652838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5465.526418] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5471.564598] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5475.224178] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5479.704100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5479.882114] Lustre: 181134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5480.278024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5483.858966] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5494.002587] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5506.021756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5506.026687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.037937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5506.037946] Lustre: Skipped 4 previous similar messages [ 5506.045531] LustreError: Skipped 1 previous similar message [ 5506.091279] Lustre: Skipped 3 previous similar messages [ 5508.163249] Lustre: server umount lustre-OST0000 complete [ 5522.195555] Lustre: server umount lustre-MDT0000 complete [ 5525.484652] LustreError: 179169:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.518453] LustreError: 179169:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5528.734884] LustreError: 179149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749573 with bad export cookie 12503646828870947107 [ 5528.740639] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.756436] LustreError: 179149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5529.883389] Lustre: server umount lustre-MDT0001 complete [ 5541.303944] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5543.970223] Key type lgssc unregistered [ 5544.255228] LNet: 182991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5544.265183] LNetError: 182991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5544.290866] LNet: Removed LNI 192.168.204.158@tcp [ 5545.089401] Key type .llcrypt unregistered [ 5545.093599] Key type ._llcrypt unregistered [ 5579.780704] Key type ._llcrypt registered [ 5579.784819] Key type .llcrypt registered [ 5579.955812] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5581.152137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5581.292049] alg: No test for adler32 (adler32-zlib) [ 5582.320109] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 5582.591205] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5584.359222] Key type lgssc registered [ 5586.237789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5594.251552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5594.280990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.995626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.792568] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5605.351276] LustreError: 184412:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.374509] LustreError: 184412:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5609.020315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5609.286815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5613.574533] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5620.016598] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5625.352552] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5631.795901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5632.121371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.126887] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5640.315115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5645.108316] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5666.275322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5666.293488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5666.308524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5666.315260] Lustre: Skipped 1 previous similar message [ 5668.118017] Lustre: server umount lustre-OST0000 complete [ 5676.001692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5676.010520] LustreError: Skipped 1 previous similar message [ 5676.015911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.026096] Lustre: Skipped 1 previous similar message [ 5676.030428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5678.006339] Lustre: server umount lustre-MDT0000 complete [ 5681.423562] LustreError: 184392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749726 with bad export cookie 512007731828450399 [ 5681.436392] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5681.706131] Lustre: server umount lustre-MDT0001 complete [ 5690.529390] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5693.053374] Key type lgssc unregistered [ 5693.339243] LNet: 187971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5693.348249] LNetError: 187971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5693.361212] LNet: Removed LNI 192.168.204.158@tcp [ 5694.235190] Key type .llcrypt unregistered [ 5694.238513] Key type ._llcrypt unregistered [ 5717.872206] Key type ._llcrypt registered [ 5717.876203] Key type .llcrypt registered [ 5717.977828] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5718.981931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5719.029639] alg: No test for adler32 (adler32-zlib) [ 5720.167899] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 5720.441303] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5722.119228] Key type lgssc registered [ 5723.419578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5731.355172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5731.370372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5732.816182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.506138] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5743.078320] LustreError: 189386:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5743.096378] LustreError: 189386:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5746.665655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.197051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5752.433359] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5761.958324] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5770.498460] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5779.254254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5779.684165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5786.146028] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5794.508540] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5802.236364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5812.704627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5812.716287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.724145] LustreError: Skipped 1 previous similar message [ 5812.726525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5812.808618] Lustre: Skipped 1 previous similar message [ 5816.760290] Lustre: lustre-OST0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 5816.767735] Lustre: Skipped 1 previous similar message [ 5817.825148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.607344] Lustre: server umount lustre-OST0000 complete [ 5828.081253] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8dc9c4468000.ost_server_uuid 50 [ 5830.464507] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5847.521272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.539802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.554831] Lustre: Skipped 2 previous similar messages [ 5851.276080] Lustre: server umount lustre-MDT0000 complete [ 5852.647033] LustreError: 190114:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.668893] LustreError: 190114:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5855.338778] LustreError: 189367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749900 with bad export cookie 12673470076402998034 [ 5855.346676] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.360696] LustreError: 189367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5855.731990] Lustre: server umount lustre-MDT0001 complete [ 5865.608306] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:58:28 (1788749908) [ 5873.030922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.509797] LustreError: 193321:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.616338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.217828] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5878.756329] LustreError: 193322:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.881888] LustreError: 193321:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.372713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.960111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5893.095174] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5901.498223] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.353496] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5915.977597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5916.352384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.386374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5923.642957] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5932.869458] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5940.720567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5941.070593] Lustre: lustre-OST0001: new disk, initializing [ 5941.074949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5941.083512] Lustre: 196166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5941.302762] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5950.017476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5950.024701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5950.090898] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5950.151761] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5961.420733] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5964.773631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5964.785909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.804460] Lustre: Skipped 1 previous similar message [ 5964.814398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.859924] Lustre: server umount lustre-OST0000 complete [ 5970.928703] LustreError: 196528:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.953827] LustreError: 196528:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5978.444641] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5978.730274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5984.643711] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5984.969756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5988.224920] LustreError: 195144:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.245772] LustreError: 195144:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 6001.642583] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.644882] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.651042] LustreError: Skipped 1 previous similar message [ 6001.653331] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6001.672667] Lustre: Skipped 2 previous similar messages [ 6001.706034] Lustre: Skipped 4 previous similar messages [ 6006.724591] Lustre: server umount lustre-OST0001 complete [ 6010.854502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6010.865601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.234892] Lustre: server umount lustre-MDT0000 complete [ 6021.072980] LustreError: 193302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750065 with bad export cookie 12673470076402999098 [ 6021.082488] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6021.089469] LustreError: 193302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6021.382124] Lustre: server umount lustre-MDT0001 complete [ 6048.004892] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:01:30 (1788750090) [ 6055.003180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.316623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6055.361102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6055.563025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.669230] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6071.159911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.322094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6071.404851] Lustre: 199619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6071.582848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6071.595818] Lustre: Skipped 3 previous similar messages [ 6077.132605] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6085.117790] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6092.611640] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6100.185081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.390252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6107.365539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6107.427384] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6116.491913] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6123.894343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.987175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6124.272400] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6124.281151] Lustre: Skipped 2 previous similar messages [ 6131.172645] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6139.786099] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6143.465453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6143.481502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.483604] LustreError: Skipped 1 previous similar message [ 6143.484558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6143.529022] Lustre: Skipped 2 previous similar messages [ 6148.823920] Lustre: server umount lustre-OST0000 complete [ 6149.604249] LustreError: 201471:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6149.620811] LustreError: 201471:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 10 previous similar messages [ 6164.961534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.979451] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.026793] Lustre: server umount lustre-OST0001 complete [ 6182.382245] Lustre: server umount lustre-MDT0000 complete [ 6187.477061] LustreError: 200365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750232 with bad export cookie 12673470076403000512 [ 6187.506450] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.913780] Lustre: server umount lustre-MDT0001 complete [ 6214.822373] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:04:17 (1788750257) [ 6232.790967] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 6242.946389] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6280.199282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.457865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6280.490413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6280.617050] Lustre: lustre-MDT0000: new disk, initializing [ 6280.795267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.811362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6285.971345] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6296.969404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.113673] Lustre: 207715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6297.184276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6297.194088] Lustre: Skipped 1 previous similar message [ 6297.272178] Lustre: lustre-MDT0001: new disk, initializing [ 6297.414769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6297.425327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6302.403759] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6312.196428] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6318.610093] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6325.516874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6325.774779] Lustre: lustre-OST0000: new disk, initializing [ 6325.778409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6325.783699] Lustre: 209685:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6330.951746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6330.971966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6331.076295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6332.034202] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6341.171628] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6346.010697] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6346.235667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6351.314084] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6351.505948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6356.963170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6356.964449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.979130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6356.979139] Lustre: Skipped 11 previous similar messages [ 6356.980119] LustreError: Skipped 3 previous similar messages [ 6357.008905] Lustre: Skipped 4 previous similar messages [ 6359.895441] Lustre: server umount lustre-OST0000 complete [ 6370.371370] Lustre: server umount lustre-MDT0000 complete [ 6372.320380] LustreError: 208846:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.347564] LustreError: 208846:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 14 previous similar messages [ 6373.990797] LustreError: 207707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750418 with bad export cookie 12673470076403002073 [ 6373.992888] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.999335] LustreError: 207707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6374.334555] Lustre: server umount lustre-MDT0001 complete [ 6382.010064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.434547] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6396.357933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6401.323712] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6409.736054] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6416.019547] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6423.242116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6430.098307] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6438.354963] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6442.972758] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6443.135587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6447.639457] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6447.920708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6452.195591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.218404] Lustre: Skipped 3 previous similar messages [ 6456.055262] Lustre: server umount lustre-OST0000 complete [ 6462.199842] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6466.399567] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6495.300513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.830042] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6510.006778] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.993153] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6518.628786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.637699] Lustre: Skipped 7 previous similar messages [ 6519.990514] Lustre: server umount lustre-OST0000 complete [ 6524.896420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6524.905535] LustreError: Skipped 1 previous similar message [ 6530.469791] Lustre: server umount lustre-MDT0000 complete [ 6534.113479] LustreError: 211906:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.128980] LustreError: 211906:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 6534.246098] LustreError: 211887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750579 with bad export cookie 12673470076403002955 [ 6534.268106] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6534.541447] Lustre: server umount lustre-MDT0001 complete [ 6542.689235] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:09:45 (1788750585) [ 6549.153487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.703582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6549.714703] Lustre: Skipped 6 previous similar messages [ 6553.782808] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6561.958526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.944273] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6575.314031] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6581.608835] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6588.827705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.597911] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6603.816395] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.344510] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6609.647741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6615.796581] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6616.111252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6624.868920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.103848] Lustre: lustre-OST0001: new disk, initializing [ 6625.107573] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6625.115315] Lustre: 221239:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6632.250333] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6633.090292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6633.102805] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6633.257583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6643.656410] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6650.571603] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6650.799592] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6657.073779] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6657.513555] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.164263] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6664.165627] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.171797] LustreError: Skipped 1 previous similar message [ 6664.200405] Lustre: Skipped 3 previous similar messages [ 6674.403409] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6674.571562] Lustre: server umount lustre-OST0001 complete [ 6681.556342] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6681.845707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6687.165836] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6687.430541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6717.106312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.535461] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6722.558258] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6722.563372] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6724.742617] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6733.696732] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6735.671064] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6744.211383] Lustre: server umount lustre-OST0001 complete [ 6776.289320] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6776.551166] Lustre: server umount lustre-OST0000 complete [ 6782.435152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6782.443022] Lustre: Skipped 17 previous similar messages [ 6787.479632] Lustre: server umount lustre-MDT0000 complete [ 6791.192689] LustreError: 217850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750836 with bad export cookie 12673470076403003907 [ 6791.204900] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6791.205542] LustreError: 217850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6797.662403] Lustre: server umount lustre-MDT0001 complete [ 6823.422687] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:14:26 (1788750866) [ 6830.865423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.188283] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6831.229483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6836.558494] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6845.872044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.028163] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6846.082995] Lustre: 226487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6851.403935] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6859.373866] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6864.933751] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6871.641789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6871.890292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6877.950939] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6885.797258] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6896.098834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.167667] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6903.216521] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6911.248397] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.277122] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6918.072461] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6918.312187] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6923.368411] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6923.570415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6929.991478] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc9d2544000.ost_server_uuid 50 [ 6931.749286] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8dc9d2544000.ost_server_uuid in IDLE state after 0 sec [ 6932.872847] Lustre: Permanently deactivating lustre-OST0001 [ 6932.905219] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6938.056965] Lustre: Permanently reactivating lustre-OST0001 [ 6938.169873] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6938.189135] Lustre: Skipped 2 previous similar messages [ 6943.200040] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6943.205922] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.217436] LustreError: Skipped 4 previous similar messages [ 6943.248977] Lustre: Skipped 7 previous similar messages [ 6949.327977] Lustre: server umount lustre-OST0001 complete [ 6953.446066] LustreError: 228336:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6953.487596] LustreError: 228336:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 40 previous similar messages [ 6960.887271] Lustre: server umount lustre-OST0000 complete [ 6971.345907] Lustre: server umount lustre-MDT0000 complete [ 6975.827251] LustreError: 226479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751020 with bad export cookie 12673470076403005195 [ 6975.828267] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.840504] LustreError: 226479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6976.210812] Lustre: server umount lustre-MDT0001 complete [ 7003.110727] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:17:25 (1788751045) [ 7011.613585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.038076] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7012.072835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7012.083964] Lustre: Skipped 2 previous similar messages [ 7018.181616] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7029.861186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.954305] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7029.980612] Lustre: 233183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7036.911203] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7047.025831] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7052.960987] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7059.721380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7059.927842] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7059.952355] Lustre: Permanently deactivating lustre-OST0000 [ 7059.962480] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7065.254120] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7066.150536] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7066.167541] LustreError: 235470:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 7072.672089] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.588397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.642542] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7079.877454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7079.884422] Lustre: Skipped 11 previous similar messages [ 7086.747689] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7088.844204] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7095.243668] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7097.579440] Lustre: Permanently reactivating lustre-OST0000 [ 7097.605946] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7097.619109] Lustre: Skipped 2 previous similar messages [ 7107.634171] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7107.634287] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 0000000020e1f57e, cur 1788751152 deadline 1788751141 last 1788751111 [ 7107.643066] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7107.668794] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7107.681486] Lustre: Skipped 1 previous similar message [ 7107.724095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7117.703496] Lustre: server umount lustre-OST0001 complete [ 7127.703893] Lustre: server umount lustre-OST0000 complete [ 7137.846861] Lustre: server umount lustre-MDT0000 complete [ 7141.336078] LustreError: 233177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751186 with bad export cookie 12673470076403006861 [ 7141.349349] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7141.651481] Lustre: server umount lustre-MDT0001 complete [ 7152.235826] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7154.971624] Key type lgssc unregistered [ 7155.281939] LNet: 238064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7155.294190] LNetError: 238064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7155.311886] LNet: Removed LNI 192.168.204.158@tcp [ 7156.256884] Key type .llcrypt unregistered [ 7156.260445] Key type ._llcrypt unregistered [ 7180.806442] Key type ._llcrypt registered [ 7180.808648] Key type .llcrypt registered [ 7180.894400] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7182.068170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7182.086415] alg: No test for adler32 (adler32-zlib) [ 7183.169662] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 7183.441463] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7185.127321] Key type lgssc registered [ 7186.229454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7194.945842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7194.962871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.373607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.291210] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7206.883516] LustreError: 239528:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.900755] LustreError: 239528:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7208.499795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.624215] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7208.661626] Lustre: 239518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7208.681306] Lustre: Permanently deactivating lustre-MDT0001 [ 7208.697446] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7209.089127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7210.723187] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7210.730269] LustreError: 239559:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7210.738962] LustreError: 239559:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7210.752909] LustreError: 239559:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7213.642428] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7221.314625] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7227.116384] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7233.841824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7234.227449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.508530] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7241.389250] LustreError: 241372:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.420403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7246.826617] LustreError: 241372:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.639154] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7251.945367] LustreError: 241370:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.417098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.662922] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7260.149780] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7260.153875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7260.386428] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7268.134625] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.787161] Lustre: Permanently reactivating lustre-MDT0001 [ 7271.800946] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7271.806188] Lustre: Skipped 2 previous similar messages [ 7273.867474] Lustre: lustre-MDT0001: Client c5ae0414-1719-4fbc-879f-4f5e57f076c8 (at 192.168.204.58@tcp) reconnecting [ 7279.295098] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7280.764749] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7285.458355] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7285.644070] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7289.371377] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7289.582957] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7291.985609] Lustre: Permanently deactivating lustre-MDT0001 [ 7292.016393] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7292.022309] Lustre: Skipped 2 previous similar messages [ 7299.684427] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7308.256660] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.260221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7308.273173] Lustre: Skipped 3 previous similar messages [ 7308.277373] Lustre: Skipped 2 previous similar messages [ 7313.380494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7313.965061] Lustre: server umount lustre-MDT0000 complete [ 7317.066788] LustreError: 239511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751361 with bad export cookie 16868142589520037149 [ 7317.081186] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.388809] Lustre: server umount lustre-MDT0001 complete [ 7326.594580] Lustre: server umount lustre-OST0000 complete [ 7335.331804] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:22:58 (1788751378) [ 7344.716360] Lustre: server umount lustre-OST0001 complete [ 7350.547415] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7358.300626] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7390.693946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.932304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7390.942566] Lustre: Skipped 2 previous similar messages [ 7390.961859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7391.048467] Lustre: lustre-MDT0000: new disk, initializing [ 7391.154772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7391.167467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7395.871829] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7405.741792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.848390] Lustre: 247822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7405.915620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7405.918516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7405.923203] Lustre: Skipped 1 previous similar message [ 7405.932249] Lustre: Skipped 3 previous similar messages [ 7405.995500] Lustre: lustre-MDT0001: new disk, initializing [ 7406.088848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7406.127568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7406.134380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7409.659791] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7418.107677] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7422.959970] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7429.299969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.566315] Lustre: lustre-OST0000: new disk, initializing [ 7429.571299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7429.579761] Lustre: 249790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7429.654306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7435.084910] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7435.799376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7435.806116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7435.924116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7443.761027] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7445.815225] LustreError: 247828:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7453.358570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7453.561343] Lustre: lustre-OST0001: new disk, initializing [ 7453.566461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7453.581237] Lustre: lustre-OST0001: Not available for connect from 192.168.204.58@tcp (not set up) [ 7453.581280] Lustre: 250903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7455.887221] LustreError: 247828:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7455.925496] LustreError: 250558:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7458.566522] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7460.915383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7460.922144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7461.012627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7465.992630] LustreError: 250558:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7466.361445] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7467.601535] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7471.588313] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.601459] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.608525] LustreError: Skipped 1 previous similar message [ 7471.624912] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7471.633819] Lustre: Skipped 1 previous similar message [ 7475.483403] Lustre: server umount lustre-OST0001 complete [ 7476.708268] LustreError: 249774:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7476.728770] LustreError: 249774:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 7486.957414] LustreError: 249775:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.977460] LustreError: 249775:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 7492.067966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7492.074216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.082576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7492.091455] Lustre: Skipped 2 previous similar messages [ 7492.095991] Lustre: Skipped 1 previous similar message [ 7505.375154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7505.516311] Lustre: server umount lustre-OST0000 complete [ 7512.550637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.554144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7512.563172] Lustre: Skipped 1 previous similar message [ 7512.570117] Lustre: Skipped 5 previous similar messages [ 7514.935732] Lustre: server umount lustre-MDT0000 complete [ 7517.664559] LustreError: 248986:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.680381] LustreError: 248986:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 7517.866956] LustreError: 247814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751562 with bad export cookie 16868142589520039753 [ 7517.869418] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7517.875367] LustreError: 247814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7524.205629] Lustre: server umount lustre-MDT0001 complete [ 7533.786968] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7536.097055] Key type lgssc unregistered [ 7536.376813] LNet: 252936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.390616] LNetError: 252936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.414645] LNet: Removed LNI 192.168.204.158@tcp [ 7537.131093] Key type .llcrypt unregistered [ 7537.132968] Key type ._llcrypt unregistered [ 7573.457579] Key type ._llcrypt registered [ 7573.459942] Key type .llcrypt registered [ 7573.565834] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7574.482792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7574.503528] alg: No test for adler32 (adler32-zlib) [ 7575.541258] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 7575.722670] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7577.375244] Key type lgssc registered [ 7578.350755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7584.756641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7584.789374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.054746] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7586.070764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7586.187711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.279300] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7597.589110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.701601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7597.740988] Lustre: 255140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7598.017049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.008952] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7608.369372] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7612.917677] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7618.652414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7618.831521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7619.076879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7624.390467] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7625.909430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7631.448643] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7682.017068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.022219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.042899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.537681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.545075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.559743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7685.054435] Lustre: server umount lustre-OST0000 complete [ 7688.804192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7699.703491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.961496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7703.033248] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7703.044339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7703.054580] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7703.064508] Lustre: Skipped 1 previous similar message [ 7705.116842] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7712.022028] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7754.215312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7754.220748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7754.222054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7754.231196] Lustre: Skipped 1 previous similar message [ 7756.739591] Lustre: server umount lustre-OST0000 complete [ 7761.888314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7761.899868] LustreError: Skipped 1 previous similar message [ 7761.907230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.923915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.929348] Lustre: Skipped 1 previous similar message [ 7766.728519] Lustre: server umount lustre-MDT0000 complete [ 7769.888170] LustreError: 255134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751814 with bad export cookie 1627618429851175650 [ 7769.893561] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7769.901847] LustreError: 255134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7770.155723] Lustre: server umount lustre-MDT0001 complete [ 7779.860785] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7782.325422] Key type lgssc unregistered [ 7782.568552] LNet: 260557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.582609] LNetError: 260557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.595780] LNet: Removed LNI 192.168.204.158@tcp [ 7783.400781] Key type .llcrypt unregistered [ 7783.407546] Key type ._llcrypt unregistered [ 7805.389584] Key type ._llcrypt registered [ 7805.394464] Key type .llcrypt registered [ 7805.526831] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7806.648459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.676195] alg: No test for adler32 (adler32-zlib) [ 7807.712894] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 7807.978347] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7809.663251] Key type lgssc registered [ 7810.693715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7817.438766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7817.459800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.799943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.293726] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7828.966115] LustreError: 261971:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.989452] LustreError: 261971:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7829.028838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.331808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7832.750879] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7838.302219] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7842.797144] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7848.570317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7848.842643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7854.019885] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7860.675706] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7863.275473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7883.744736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.748664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.760485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.768666] Lustre: Skipped 1 previous similar message [ 7883.783396] Lustre: Skipped 1 previous similar message [ 7888.871157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7888.876824] Lustre: Skipped 1 previous similar message [ 7889.260494] Lustre: server umount lustre-OST0000 complete [ 7893.988957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7893.992399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7894.003687] Lustre: Skipped 1 previous similar message [ 7894.011820] Lustre: Skipped 1 previous similar message [ 7899.104544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.112664] Lustre: Skipped 1 previous similar message [ 7904.224694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.301417] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7907.479390] Lustre: server umount lustre-MDT0000 complete [ 7909.353354] LustreError: 261965:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.383119] LustreError: 261965:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7910.417321] LustreError: 261949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751955 with bad export cookie 12743850493948385993 [ 7910.430382] LustreError: 261949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7910.430594] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.649270] Lustre: server umount lustre-MDT0001 complete [ 7919.248128] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7921.470376] Key type lgssc unregistered [ 7921.695755] LNet: 266278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7921.704650] LNetError: 266278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7921.722522] LNet: Removed LNI 192.168.204.158@tcp [ 7922.331532] Key type .llcrypt unregistered [ 7922.341175] Key type ._llcrypt unregistered [ 7936.370584] Key type ._llcrypt registered [ 7936.373534] Key type .llcrypt registered [ 7936.471961] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7937.096236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7937.170047] alg: No test for adler32 (adler32-zlib) [ 7938.137782] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 7938.324100] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7939.959333] Key type lgssc registered [ 7940.913646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7946.573776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7946.607168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.972372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.079787] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7957.422432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.486276] LustreError: 267476:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.509859] LustreError: 267476:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7957.753438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7961.218512] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7967.579910] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7972.171794] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7977.794993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7978.088398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.733705] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7988.208500] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.424127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8003.051375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.054615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.078585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8006.629279] Lustre: server umount lustre-OST0000 complete [ 8013.284383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8013.294582] Lustre: Skipped 1 previous similar message [ 8013.308746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8013.315617] Lustre: Skipped 2 previous similar messages [ 8016.009480] Lustre: server umount lustre-MDT0000 complete [ 8018.400555] LustreError: 268956:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.416954] LustreError: 268956:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8018.585844] LustreError: 267454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752063 with bad export cookie 2886453793642131545 [ 8018.594670] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8018.820128] Lustre: server umount lustre-MDT0001 complete [ 8027.158462] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 8029.310060] Key type lgssc unregistered [ 8029.616509] LNet: 271151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8029.620981] LNetError: 271151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8029.649864] LNet: Removed LNI 192.168.204.158@tcp [ 8030.453793] Key type .llcrypt unregistered [ 8030.455774] Key type ._llcrypt unregistered [ 8048.171976] Key type ._llcrypt registered [ 8048.175220] Key type .llcrypt registered [ 8048.252590] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8048.706154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8048.757962] alg: No test for adler32 (adler32-zlib) [ 8049.732175] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 8049.921629] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 8051.551317] Key type lgssc registered [ 8052.366697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.851987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8057.864935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.116429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.114177] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8068.018558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.298912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8071.297500] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8076.285199] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8080.351333] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8085.258323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8085.534101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8090.406727] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8095.696861] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8105.005930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8124.385731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8124.394951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.405604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8125.410166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8125.412288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.413682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8129.590425] Lustre: server umount lustre-OST0000 complete [ 8135.136783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8135.143068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.157702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8135.161642] Lustre: Skipped 1 previous similar message [ 8138.650053] Lustre: server umount lustre-MDT0000 complete [ 8140.777624] LustreError: 272540:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.792263] LustreError: 272540:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 8141.581519] LustreError: 272526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752186 with bad export cookie 18057042937185280961 [ 8141.583047] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8141.589779] LustreError: 272526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8141.786706] Lustre: server umount lustre-MDT0001 complete [ 8149.837509] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 8151.770577] Key type lgssc unregistered [ 8151.985811] LNet: 276870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8151.992711] LNetError: 276870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8152.006209] LNet: Removed LNI 192.168.204.158@tcp [ 8152.683137] Key type .llcrypt unregistered [ 8152.686549] Key type ._llcrypt unregistered [ 8167.700288] Key type ._llcrypt registered [ 8167.701695] Key type .llcrypt registered [ 8167.800371] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8168.515392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8168.605441] alg: No test for adler32 (adler32-zlib) [ 8169.674509] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 8169.864123] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 8171.543222] Key type lgssc registered [ 8172.496757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8178.869510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8178.883423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.241850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.852346] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8190.438597] LustreError: 278102:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.455189] LustreError: 278102:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8191.416668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8191.709046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8195.406399] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8202.007667] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8206.808797] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8212.476504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.795943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.819374] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8223.778428] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8226.109081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8236.514560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.520088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.528556] Lustre: Skipped 1 previous similar message [ 8241.633829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.644360] Lustre: Skipped 2 previous similar messages [ 8242.351788] Lustre: server umount lustre-OST0000 complete [ 8246.754056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.754697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.765594] Lustre: Skipped 1 previous similar message [ 8246.773626] Lustre: Skipped 1 previous similar message [ 8251.494675] Lustre: server umount lustre-MDT0000 complete [ 8251.875391] LustreError: 278097:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.889277] LustreError: 278097:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8254.342954] LustreError: 281051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752299 with bad export cookie 3790670456427409655 [ 8254.343593] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.359744] LustreError: 281051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8254.605070] Lustre: server umount lustre-MDT0001 complete [ 8263.515865] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 8265.688987] Key type lgssc unregistered [ 8265.984752] LNet: 281803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8265.993892] LNetError: 281803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8266.022807] LNet: Removed LNI 192.168.204.158@tcp [ 8266.660216] Key type .llcrypt unregistered [ 8266.664161] Key type ._llcrypt unregistered [ 8276.197693] Key type ._llcrypt registered [ 8276.200681] Key type .llcrypt registered [ 8276.302784] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8289.389973] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 8301.403175] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8301.972924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8302.062600] alg: No test for adler32 (adler32-zlib) [ 8303.115436] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 8303.332041] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 8304.992692] Key type lgssc registered [ 8305.827698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8333.064254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8333.080280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.289937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8334.322441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8334.407988] Lustre: lustre-MDT0000: new disk, initializing [ 8334.484885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.497871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8338.072609] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8346.961985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.075387] Lustre: 285329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8347.126240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8347.130964] Lustre: Skipped 1 previous similar message [ 8347.194146] Lustre: lustre-MDT0001: new disk, initializing [ 8347.289221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.321132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8347.338030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8351.176385] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8360.299599] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8365.247736] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8371.006421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.282826] Lustre: lustre-OST0000: new disk, initializing [ 8371.286819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8371.294730] Lustre: 287296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8371.394485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.622645] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8379.970200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8379.986152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8380.064599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8383.768593] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8387.177333] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8387.354683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8390.514359] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8390.653477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8395.746785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8395.759424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8395.775209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8395.781313] Lustre: Skipped 1 previous similar message [ 8398.217522] Lustre: server umount lustre-OST0000 complete [ 8403.938806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8403.948045] LustreError: Skipped 1 previous similar message [ 8403.958613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.971153] Lustre: Skipped 1 previous similar message [ 8403.979266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.987477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.995120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8407.918315] Lustre: server umount lustre-MDT0000 complete [ 8410.660338] LustreError: 287295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752455 with bad export cookie 4772194215904841044 [ 8410.661627] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8410.670244] LustreError: 287295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8410.945661] Lustre: server umount lustre-MDT0001 complete [ 8416.916212] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:41:00 (1788752460) [ 8421.742832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.180110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8425.534809] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8427.492215] LustreError: 289706:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.509939] LustreError: 289706:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8431.592578] LustreError: 289705:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8432.414806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.743599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8436.497646] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8441.802678] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8445.947306] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8451.795793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8452.103202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.810320] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8462.630559] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8476.654370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8476.658592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.672859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.678778] Lustre: Skipped 2 previous similar messages [ 8481.487894] Lustre: server umount lustre-OST0000 complete [ 8486.883112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.888194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8486.893629] Lustre: Skipped 2 previous similar messages [ 8486.901672] Lustre: Skipped 1 previous similar message [ 8490.276225] Lustre: server umount lustre-MDT0000 complete [ 8492.000458] LustreError: 289706:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.022458] LustreError: 289706:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8493.161572] LustreError: 289685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752537 with bad export cookie 4772194215904841919 [ 8493.167081] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8493.174175] LustreError: 289685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8493.454404] Lustre: server umount lustre-MDT0001 complete [ 8501.565291] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 8503.741287] Key type lgssc unregistered [ 8503.983629] LNet: 293265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8503.993940] LNetError: 293265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8504.012524] LNet: Removed LNI 192.168.204.158@tcp [ 8504.794800] Key type .llcrypt unregistered [ 8504.807710] Key type ._llcrypt unregistered [ 8527.577989] Key type ._llcrypt registered [ 8527.581340] Key type .llcrypt registered [ 8527.702103] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 8555.610456] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8557.260294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8557.508980] alg: No test for adler32 (adler32-zlib) [ 8559.273581] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 8559.730747] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 8561.575293] Key type lgssc registered [ 8563.539858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8620.636591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8620.663646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.108704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8622.201946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8622.316967] Lustre: lustre-MDT0000: new disk, initializing [ 8622.405123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.427473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8627.070979] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8637.121190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.277624] Lustre: 297089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8637.332652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8637.336954] Lustre: Skipped 1 previous similar message [ 8637.395378] Lustre: lustre-MDT0001: new disk, initializing [ 8637.501608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8637.542583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8637.551932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8642.027210] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8650.324308] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8655.819887] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8662.939497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.229869] Lustre: lustre-OST2710: new disk, initializing [ 8663.232915] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8663.242425] Lustre: 299058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8663.339281] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8669.899740] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8672.356479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8672.367056] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8672.485755] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8680.454602] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8687.805816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8687.983909] Lustre: lustre-OST03e8: new disk, initializing [ 8687.989247] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8687.998528] Lustre: 300123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8688.088607] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8694.915286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8694.929491] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8695.030092] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8695.697120] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8704.676199] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8712.759863] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8712.963962] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8717.673804] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8717.936314] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8722.816185] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8723.072895] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8728.018778] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8728.353923] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8734.691127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8734.703126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8734.734227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8735.229225] Lustre: server umount lustre-MDT0000 complete [ 8736.236762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.256100] Lustre: Skipped 1 previous similar message [ 8739.024234] LustreError: 297083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752783 with bad export cookie 5104746497612798141 [ 8739.026318] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8739.034374] LustreError: 297083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8739.398981] Lustre: server umount lustre-MDT0001 complete [ 8753.497992] Lustre: server umount lustre-OST2710 complete [ 8768.130173] Lustre: server umount lustre-OST03e8 complete [ 8788.972690] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 8801.744281] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8841.968776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.371913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8842.408807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8842.538463] Lustre: lustre-MDT0000: new disk, initializing [ 8842.643806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8842.660475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8847.919840] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8861.994604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.279452] Lustre: 305376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8862.413521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8862.422921] Lustre: Skipped 1 previous similar message [ 8862.566176] Lustre: lustre-MDT0001: new disk, initializing [ 8862.693155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8862.728791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8862.738684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8867.985431] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8879.628365] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8886.583846] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8893.925677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8894.239924] Lustre: lustre-OST0000: new disk, initializing [ 8894.247396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8894.261313] Lustre: 307344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8894.400533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.682797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8900.696683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8900.804420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8901.135225] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8911.439162] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8915.997505] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8916.219308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8921.112765] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8921.435539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8926.694435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8926.697765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8926.726548] Lustre: Skipped 1 previous similar message [ 8926.750788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8926.759716] Lustre: Skipped 1 previous similar message [ 8929.923266] Lustre: server umount lustre-OST0000 complete [ 8934.374489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8934.381531] LustreError: Skipped 1 previous similar message [ 8934.386713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.404748] Lustre: Skipped 1 previous similar message [ 8934.415886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8936.933112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8936.945416] Lustre: Skipped 1 previous similar message [ 8940.390326] Lustre: server umount lustre-MDT0000 complete [ 8942.054687] LustreError: 307628:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8942.099776] LustreError: 307628:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 8944.391707] LustreError: 306172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752989 with bad export cookie 5104746497612799821 [ 8944.395685] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8944.412445] LustreError: 306172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8945.163527] Lustre: server umount lustre-MDT0001 complete [ 8955.298102] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:49:58 (1788752998) [ 8957.464492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8959.496664] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:50:02 (1788753002) [ 8963.684103] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8995.672966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8995.967921] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8995.991901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8996.201664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.035762] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9009.770168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.894205] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9009.918921] Lustre: 311200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9014.788649] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9036.520514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9036.749460] Lustre: Denying initial registration attempt from nid 192.168.204.158@tcp, specified as failover [ 9036.760442] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9036.772634] LustreError: 312654:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9036.777609] LustreError: 312654:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9036.784983] LustreError: 312654:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9036.852557] Lustre: server umount lustre-OST0000 complete [ 9036.855468] LustreError: 312654:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9045.985983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9045.994649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9046.009227] Lustre: Skipped 1 previous similar message [ 9046.016312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9056.747352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9056.763345] Lustre: Skipped 5 previous similar messages [ 9057.247167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9057.468590] Lustre: server umount lustre-MDT0000 complete [ 9061.408987] LustreError: 311194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753106 with bad export cookie 5104746497612800703 [ 9061.421523] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.731772] Lustre: server umount lustre-MDT0001 complete [ 9073.425035] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 9076.144696] Key type lgssc unregistered [ 9076.501789] LNet: 313776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9076.515075] LNetError: 313776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9076.537351] LNet: Removed LNI 192.168.204.158@tcp [ 9077.403698] Key type .llcrypt unregistered [ 9077.411651] Key type ._llcrypt unregistered [ 9088.376549] Key type ._llcrypt registered [ 9088.378803] Key type .llcrypt registered [ 9088.466970] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9089.358829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9089.398902] alg: No test for adler32 (adler32-zlib) [ 9090.603909] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 9090.862916] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 9092.535286] Key type lgssc registered [ 9093.720873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9140.827737] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9152.771922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9152.797385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.207885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9154.233674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9154.447568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9160.293569] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9171.352509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.560957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9171.629623] Lustre: 316541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9172.001322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9177.187872] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9185.931648] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9192.655603] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9212.604612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9212.952528] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9213.256436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9220.048153] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9227.889268] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9230.304841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9230.321576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.354075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9230.821674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9230.849240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.884765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9235.936977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9241.071435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9241.086708] Lustre: Skipped 2 previous similar messages [ 9244.129044] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9244.389865] Lustre: server umount lustre-OST0000 complete [ 9248.736568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9248.741405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9248.752071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9251.303858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9254.193580] Lustre: server umount lustre-MDT0000 complete [ 9257.967139] LustreError: 316533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753302 with bad export cookie 15761465723655522135 [ 9257.970496] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9257.974456] LustreError: 316533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9258.289390] Lustre: server umount lustre-MDT0001 complete [ 9269.905902] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 9273.235146] Key type lgssc unregistered [ 9273.664771] LNet: 320116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9273.682614] LNetError: 320116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9273.721022] LNet: Removed LNI 192.168.204.158@tcp [ 9274.939799] Key type .llcrypt unregistered [ 9274.942155] Key type ._llcrypt unregistered [ 9303.902552] Key type ._llcrypt registered [ 9303.906338] Key type .llcrypt registered [ 9304.017681] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9304.903945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9304.954219] alg: No test for adler32 (adler32-zlib) [ 9306.057681] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 9306.360129] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 9308.103454] Key type lgssc registered [ 9309.249118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9317.125199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9317.148167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.642370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9324.031802] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9329.130131] LustreError: 321530:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.141638] LustreError: 321530:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9334.249261] LustreError: 321531:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.879921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.243813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.145582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9349.669822] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9355.886151] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9364.972578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9365.440258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9372.293896] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9381.525664] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.829467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9401.840298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.871437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9402.790864] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 9404.387166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.403758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.419600] Lustre: Skipped 1 previous similar message [ 9406.176267] Lustre: server umount lustre-MDT0000 complete [ 9407.863485] LustreError: 321527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.114512] LustreError: 321511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753454 with bad export cookie 8873346331815202893 [ 9410.116868] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9410.125781] LustreError: 321511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9412.997727] LustreError: 322628:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9413.034118] LustreError: 322628:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 9413.047751] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (stopping) [ 9413.071026] Lustre: Skipped 2 previous similar messages [ 9414.629513] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9416.581613] Lustre: server umount lustre-MDT0001 complete [ 9423.497524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9433.285343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.463349] LustreError: 325135:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9434.474844] LustreError: 325135:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008de15a9d x1875643833823104/t0(0) o250->MGC192.168.204.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788753479 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9434.512531] LustreError: 325135:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9435.548382] LustreError: 323364:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.588315] LustreError: 323364:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9435.686092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9435.795112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9441.707688] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9444.838592] LustreError: 325147:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.856927] LustreError: 325147:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 9445.872880] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9451.447604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.918623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9456.620045] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9457.128679] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9464.912656] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9466.775549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9472.387326] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9474.076035] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9481.697687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9481.706348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9481.723086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9481.729985] Lustre: Skipped 2 previous similar messages [ 9482.740466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9486.620429] Lustre: server umount lustre-OST0000 complete [ 9492.967712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9492.971697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.974123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9492.974131] Lustre: Skipped 2 previous similar messages [ 9493.024197] Lustre: Skipped 2 previous similar messages [ 9505.247305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9505.571863] Lustre: server umount lustre-MDT0000 complete [ 9507.298255] LustreError: 325152:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9507.331681] LustreError: 325152:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 9509.532177] LustreError: 321510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753554 with bad export cookie 8873346331815210516 [ 9509.544606] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9509.546863] LustreError: 321510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9510.280128] Lustre: server umount lustre-MDT0001 complete [ 9523.247517] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 9526.361894] Key type lgssc unregistered [ 9526.676523] LNet: 327822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9526.681974] LNetError: 327822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9526.706983] LNet: Removed LNI 192.168.204.158@tcp [ 9527.834107] Key type .llcrypt unregistered [ 9527.837361] Key type ._llcrypt unregistered [ 9558.972727] Key type ._llcrypt registered [ 9558.977918] Key type .llcrypt registered [ 9559.149570] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9560.650355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9560.697845] alg: No test for adler32 (adler32-zlib) [ 9561.912925] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 9562.254477] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 9563.959277] Key type lgssc registered [ 9565.258739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9573.590790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9573.608860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9575.116954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.830383] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9584.614135] LustreError: 329240:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9584.654399] LustreError: 329240:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9589.216410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.823559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9596.280863] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9606.835637] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9615.150706] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9619.428117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.438416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.545410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.552751] Lustre: Skipped 2 previous similar messages [ 9625.808382] Lustre: server umount lustre-MDT0000 complete [ 9629.668650] LustreError: 329968:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.679580] LustreError: 329968:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9630.716461] LustreError: 329222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753675 with bad export cookie 8377051578329803998 [ 9630.732782] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9631.147227] Lustre: server umount lustre-MDT0001 complete [ 9649.839412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.248502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9650.279690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9650.452181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.278567] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9665.879278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.992381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9666.036984] Lustre: 331843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9666.266481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9671.775746] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9681.382538] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9689.144544] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9700.019529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9700.381751] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9700.405206] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9700.415550] LustreError: 333727:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9700.437713] LustreError: 333727:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9700.457421] LustreError: 333727:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9700.572838] Lustre: server umount lustre-OST0000 complete [ 9700.583800] LustreError: 333727:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9712.253939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9712.522906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9712.748739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.459668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9720.128678] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9729.016462] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9737.909581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.059843] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9738.167630] Lustre: lustre-OST0001: new disk, initializing [ 9738.173403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9738.179794] Lustre: 335267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9738.295273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9745.747907] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9748.176177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9748.211078] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9748.324375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9756.010512] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9763.815655] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.837543] Lustre: Skipped 1 previous similar message [ 9763.854971] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9763.855701] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9766.382522] Lustre: server umount lustre-OST0001 complete [ 9768.951951] LustreError: 334194:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9768.969705] LustreError: 334194:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9770.976714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9770.985549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.012937] Lustre: Skipped 1 previous similar message [ 9771.020242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9771.036189] Lustre: Skipped 1 previous similar message [ 9774.063693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.065431] LustreError: 334195:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.111047] LustreError: 334195:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9779.183567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.191679] Lustre: Skipped 3 previous similar messages [ 9779.197623] LustreError: 334196:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9779.210263] LustreError: 334196:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9785.311665] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9785.629654] Lustre: server umount lustre-OST0000 complete [ 9794.534125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.535902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9794.560196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9794.571341] Lustre: Skipped 3 previous similar messages [ 9796.656157] Lustre: server umount lustre-MDT0000 complete [ 9799.650819] LustreError: 331854:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.674719] LustreError: 331854:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 9800.929427] LustreError: 331833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753845 with bad export cookie 8377051578329804551 [ 9800.945338] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.952477] LustreError: 331833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9801.465393] Lustre: server umount lustre-MDT0001 complete [ 9813.812889] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 9817.015869] Key type lgssc unregistered [ 9817.481575] LNet: 337229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9817.496221] LNetError: 337229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9817.527914] LNet: Removed LNI 192.168.204.158@tcp [ 9818.457843] Key type .llcrypt unregistered [ 9818.460908] Key type ._llcrypt unregistered [ 9903.780337] Key type ._llcrypt registered [ 9903.783521] Key type .llcrypt registered [ 9903.929386] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 9924.439553] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9925.635793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9925.771086] alg: No test for adler32 (adler32-zlib) [ 9926.899465] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [ 9927.187529] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 9928.903280] Key type lgssc registered [ 9930.328469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9974.142539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9974.164162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.571837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9975.603494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9975.746087] Lustre: lustre-MDT0000: new disk, initializing [ 9975.859493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9975.895640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9981.843560] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9994.528796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.706160] Lustre: 342164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9994.799956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9994.804075] Lustre: Skipped 1 previous similar message [ 9994.929454] Lustre: lustre-MDT0001: new disk, initializing [ 9995.032913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9995.083428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9995.098095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10000.633343] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10011.172816] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10017.401143] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10024.698915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10025.000531] Lustre: lustre-OST0000: new disk, initializing [10025.005104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10025.013871] Lustre: 344133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10025.130586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10032.218551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10032.223839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10032.316124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10032.632490] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10042.934728] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10049.083424] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10049.435271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10054.687802] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10054.920944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10058.216177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10058.230983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.260114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10058.265031] Lustre: Skipped 1 previous similar message [10063.597477] Lustre: server umount lustre-OST0000 complete [10068.454759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.458366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.473242] Lustre: Skipped 2 previous similar messages [10068.491375] Lustre: Skipped 1 previous similar message [10073.568949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.579979] Lustre: Skipped 1 previous similar message [10074.037402] Lustre: server umount lustre-MDT0000 complete [10077.987319] LustreError: 342156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754122 with bad export cookie 4243229214929695479 [10078.009904] LustreError: 342156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10078.017344] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10078.574777] Lustre: server umount lustre-MDT0001 complete [10088.623828] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:08:51 (1788754131) [10097.266520] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:08:59 (1788754139) [10107.513773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.050023] LustreError: 346890:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.063722] LustreError: 346890:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10108.134837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.329604] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10113.519111] LustreError: 346891:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.605897] LustreError: 346890:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.657690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.056780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.474318] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10136.376278] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10142.464991] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10149.814775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10150.286568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10157.177694] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10165.455981] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10174.641119] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10176.893766] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10179.281320] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10181.888771] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10184.513600] Lustre: Failing over lustre-MDT0000 [10185.044654] Lustre: server umount lustre-MDT0000 complete [10187.237783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.239904] LustreError: 346887:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.254164] Lustre: Skipped 1 previous similar message [10187.292974] LustreError: 346887:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10192.355795] LustreError: 346890:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.373120] LustreError: 346890:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10202.621535] LustreError: 346891:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10202.654602] LustreError: 346891:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [10203.679603] Lustre: 340387:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754232/real 1788754232] req@000000006925a280 x1875644485019136/t0(0) o400->MGC192.168.204.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788754248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10203.717263] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10205.601238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.745771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10213.823966] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10214.274281] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10218.999811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10219.070159] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10219.073438] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10219.150970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10230.654734] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10232.110599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10236.206821] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10238.243780] Lustre: Failing over lustre-MDT0000 [10238.521716] Lustre: server umount lustre-MDT0000 complete [10239.456729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10239.461542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.471365] LustreError: Skipped 1 previous similar message [10239.480300] LustreError: 346891:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10239.492883] Lustre: Skipped 2 previous similar messages [10239.552783] LustreError: 346891:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 12 previous similar messages [10255.839497] Lustre: 340389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754284/real 1788754284] req@0000000062c27b24 x1875644485040000/t0(0) o400->MGC192.168.204.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788754300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10255.874165] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.855503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.089924] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3ae2fa7d4871bd93 [10266.112783] Lustre: MGC192.168.204.158@tcp: Connection restored to 0@lo (at 0@lo) [10266.121259] Lustre: Skipped 2 previous similar messages [10266.430923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.486907] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10266.522883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10271.695088] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10271.768985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10271.837922] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10271.890567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10281.367665] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10283.269123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10292.193843] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.195882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10292.217537] Lustre: Skipped 2 previous similar messages [10292.232072] Lustre: Skipped 2 previous similar messages [10296.063912] Lustre: server umount lustre-MDT0000 complete [10297.315646] LustreError: 346886:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.341799] LustreError: 346886:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 26 previous similar messages [10300.297569] LustreError: 346870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754345 with bad export cookie 4243229214929698195 [10300.299621] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10300.308681] LustreError: 346870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10300.719959] Lustre: server umount lustre-MDT0001 complete [10315.930424] Lustre: server umount lustre-OST0000 complete [10332.405669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.113629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10338.254416] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10347.621032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.287666] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10361.032177] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10366.862510] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10374.217490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10377.784790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10381.161470] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10388.874074] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10403.296598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10403.307249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.317427] LustreError: Skipped 1 previous similar message [10403.319655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10403.355978] Lustre: Skipped 2 previous similar messages [10406.091748] Lustre: server umount lustre-OST0000 complete [10413.547035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10413.560683] Lustre: Skipped 1 previous similar message [10417.087108] Lustre: server umount lustre-MDT0000 complete [10418.662736] LustreError: 353237:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.673082] LustreError: 353237:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [10421.132717] LustreError: 353218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754465 with bad export cookie 4243229214929698944 [10421.152862] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10421.155621] LustreError: 353218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10421.608879] Lustre: server umount lustre-MDT0001 complete [10434.023796] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [10437.010476] Key type lgssc unregistered [10437.309220] LNet: 356769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10437.322428] LNetError: 356769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10437.358511] LNet: Removed LNI 192.168.204.158@tcp [10438.354114] Key type .llcrypt unregistered [10438.356277] Key type ._llcrypt unregistered [10465.587446] Key type ._llcrypt registered [10465.590406] Key type .llcrypt registered [10465.691776] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10466.942435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10466.977114] alg: No test for adler32 (adler32-zlib) [10468.086318] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [10468.413433] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [10470.112172] Key type lgssc registered [10471.454484] Lustre: Echo OBD driver; http://www.lustre.org/ [10479.255257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10479.282208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.637528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10485.440351] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10490.859662] LustreError: 358281:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.894741] LustreError: 358281:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10493.621565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.972909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10498.469240] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10505.217140] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10511.764238] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10518.773470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10519.214267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.391898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10525.466772] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10533.228280] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10549.190487] Lustre: server umount lustre-MDT0000 complete [10549.250636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.253091] LustreError: 359289:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.268281] Lustre: Skipped 1 previous similar message [10549.286633] LustreError: 359289:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10552.975475] LustreError: 358263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754597 with bad export cookie 755427656561703271 [10552.990734] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10552.997562] LustreError: 358263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10553.386496] Lustre: server umount lustre-MDT0001 complete [10567.325876] Lustre: server umount lustre-OST0000 complete [10580.359732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.764203] LustreError: 362014:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10580.872471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.254213] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10586.090402] LustreError: 362015:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.179813] LustreError: 362014:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10594.625504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.983312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10600.375073] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10608.419789] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10614.335305] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10621.348950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.796371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10625.925632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10628.935965] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10636.879654] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10649.570192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10649.581321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10649.595898] Lustre: Skipped 2 previous similar messages [10654.689873] LustreError: 362014:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10654.715488] LustreError: 362014:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10654.827929] Lustre: server umount lustre-MDT0000 complete [10658.991133] LustreError: 364695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754703 with bad export cookie 755427656561704384 [10659.004714] LustreError: 364695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10659.011116] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10659.425832] Lustre: server umount lustre-MDT0001 complete [10674.288674] Lustre: server umount lustre-OST0000 complete [10691.956962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.534916] LustreError: 365927:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.638741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10697.508773] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10707.209319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.626141] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10721.005621] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10727.278802] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10734.443061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.741993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10734.750951] Lustre: Skipped 1 previous similar message [10738.857198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10741.510984] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10749.992170] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10758.632371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10758.634195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10758.651329] Lustre: Skipped 1 previous similar message [10758.655097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10758.680287] Lustre: Skipped 3 previous similar messages [10761.111264] Lustre: server umount lustre-OST0000 complete [10768.872686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.874619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10768.884899] Lustre: Skipped 2 previous similar messages [10768.904577] Lustre: Skipped 1 previous similar message [10771.453830] Lustre: server umount lustre-MDT0000 complete [10773.997412] LustreError: 365927:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10774.021538] LustreError: 365927:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10775.656458] LustreError: 365909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754820 with bad export cookie 755427656561705406 [10775.658087] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10775.683933] LustreError: 365909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10776.185076] Lustre: server umount lustre-MDT0001 complete [10789.237714] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [10791.949546] Key type lgssc unregistered [10792.381120] LNet: 369460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10792.403223] LNetError: 369460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10792.426795] LNet: Removed LNI 192.168.204.158@tcp [10793.328716] Key type .llcrypt unregistered [10793.333794] Key type ._llcrypt unregistered [10822.883434] Key type ._llcrypt registered [10822.886200] Key type .llcrypt registered [10822.999434] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10824.229727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10824.259374] alg: No test for adler32 (adler32-zlib) [10825.622180] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [10826.034887] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [10827.791645] Key type lgssc registered [10829.182660] Lustre: Echo OBD driver; http://www.lustre.org/ [10838.259276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10838.286848] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10838.295214] LustreError: 370897:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10838.321874] LustreError: 370897:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10838.338149] LustreError: 370897:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10838.344399] LustreError: 370897:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10838.356617] LustreError: 370897:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10846.841640] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10846.855576] LustreError: 371295:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10846.909580] LustreError: 371295:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10846.918092] LustreError: 371295:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10846.930971] LustreError: 371295:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10846.946233] LustreError: 371295:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10867.006858] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [10869.832260] Key type lgssc unregistered [10870.103981] LNet: 372299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10870.111368] LNetError: 372299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10870.124906] LNet: Removed LNI 192.168.204.158@tcp [10870.970474] Key type .llcrypt unregistered [10870.975649] Key type ._llcrypt unregistered [10892.124897] Key type ._llcrypt registered [10892.126829] Key type .llcrypt registered [10892.247562] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [10914.826392] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10915.967437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10916.129046] alg: No test for adler32 (adler32-zlib) [10917.476427] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [10917.771968] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [10919.431242] Key type lgssc registered [10921.112881] Lustre: Echo OBD driver; http://www.lustre.org/ [10962.829888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10962.881324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10964.223483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10964.286618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10964.461876] Lustre: lustre-MDT0000: new disk, initializing [10964.584916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10964.599906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10969.238552] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10979.923808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.062695] Lustre: 375455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10980.138263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10980.142337] Lustre: Skipped 1 previous similar message [10980.224567] Lustre: lustre-MDT0001: new disk, initializing [10980.348403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10980.403762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10980.421187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10985.136943] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10995.289277] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11002.022886] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11009.803704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.059422] Lustre: lustre-OST0000: new disk, initializing [11010.063981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11010.069659] Lustre: 377424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11010.159668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11016.579759] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11019.322512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11019.338310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11019.423356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11027.798469] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11032.585446] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11032.876454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11037.918146] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11038.177477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11045.348153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11045.361052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11045.369640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11045.398253] Lustre: Skipped 1 previous similar message [11045.414679] Lustre: Skipped 1 previous similar message [11046.970470] Lustre: server umount lustre-OST0000 complete [11052.004182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11052.017238] LustreError: Skipped 1 previous similar message [11052.023942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11052.054791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11055.585806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11055.593250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11055.640604] Lustre: Skipped 1 previous similar message [11060.707441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11060.716632] Lustre: Skipped 1 previous similar message [11065.312885] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11065.631058] Lustre: server umount lustre-MDT0000 complete [11069.570336] LustreError: 377423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755114 with bad export cookie 3270721224583043368 [11069.587072] LustreError: 377423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11069.589852] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11069.935690] Lustre: server umount lustre-MDT0001 complete [11080.268692] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11082.100167] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:25:24 (1788755124) [11089.612268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11090.203178] LustreError: 379888:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.224830] LustreError: 379888:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11090.289294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11095.522523] LustreError: 379889:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.672763] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11100.644608] LustreError: 379888:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.998093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.370868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11110.539959] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11119.693785] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11127.196952] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11135.802145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11136.339241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11144.140632] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11153.315182] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11161.312480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11161.468210] Lustre: lustre-OST0001: new disk, initializing [11161.475606] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11161.481729] Lustre: 382726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11161.562164] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11168.764577] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11169.388513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11169.400789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11169.518156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11178.229774] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11183.016151] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11183.026696] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11183.028671] Lustre: Skipped 1 previous similar message [11187.870753] Lustre: server umount lustre-OST0001 complete [11188.212627] LustreError: 381713:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11188.230491] LustreError: 381713:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11193.312806] LustreError: 381712:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11193.335420] LustreError: 381712:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [11203.579434] LustreError: 381713:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.628427] LustreError: 381713:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11207.137444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11207.138878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11207.150777] LustreError: Skipped 1 previous similar message [11207.155739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11207.167224] Lustre: Skipped 1 previous similar message [11207.180729] Lustre: Skipped 3 previous similar messages [11211.512585] Lustre: server umount lustre-OST0000 complete [11217.894232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11217.909961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.873588] Lustre: server umount lustre-MDT0000 complete [11224.035317] LustreError: 379883:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.068618] LustreError: 379883:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11226.629802] LustreError: 379868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755271 with bad export cookie 3270721224583044250 [11226.644185] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11226.650507] LustreError: 379868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11233.112268] Lustre: server umount lustre-MDT0001 complete [11247.259551] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [11250.548303] Key type lgssc unregistered [11250.898435] LNet: 384714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11250.907307] LNetError: 384714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11250.926207] LNet: Removed LNI 192.168.204.158@tcp [11251.958884] Key type .llcrypt unregistered [11251.961330] Key type ._llcrypt unregistered [11302.484768] Key type ._llcrypt registered [11302.487589] Key type .llcrypt registered [11302.604509] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [11303.963374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11304.016549] alg: No test for adler32 (adler32-zlib) [11305.138660] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [11305.437761] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [11307.103204] Key type lgssc registered [11308.243930] Lustre: Echo OBD driver; http://www.lustre.org/ [11316.051792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11316.085723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11317.421442] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11317.452678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11317.572611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11322.039587] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11330.634586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11330.795197] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11330.846860] Lustre: 387508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11331.125667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11335.400608] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11342.733190] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11348.335959] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11351.527278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11351.527339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11351.544120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11351.570499] Lustre: Skipped 1 previous similar message [11356.722832] Lustre: server umount lustre-MDT0000 complete [11361.269491] LustreError: 387502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755406 with bad export cookie 15979283911118778840 [11361.270623] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11361.289033] LustreError: 387502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11361.563050] Lustre: server umount lustre-MDT0001 complete [11372.739916] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:30:15 (1788755415) [11380.806149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11381.449910] LustreError: 390021:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11381.470274] LustreError: 390021:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11381.572671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11386.453551] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11386.850359] LustreError: 390022:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11391.968129] LustreError: 390021:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.855568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.145840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11399.914281] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11407.556487] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11412.832179] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11419.078789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11419.265730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11419.441949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11425.994705] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11433.690850] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11441.717432] Lustre: Permanently deactivating lustre-OST0000 [11441.732707] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11451.457115] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11452.934684] LustreError: 393057:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11452.945307] LustreError: 393057:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11452.952758] LustreError: 393057:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11456.316057] LustreError: 393105:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11456.328645] LustreError: 393105:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11464.526200] Lustre: server umount lustre-OST0000 complete [11467.249863] LustreError: 393355:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11467.257108] LustreError: 393355:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11470.314342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11470.337349] Lustre: Skipped 1 previous similar message [11470.343578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.426074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.434323] Lustre: Skipped 2 previous similar messages [11478.617069] Lustre: server umount lustre-MDT0000 complete [11480.550861] LustreError: 390018:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11480.589239] LustreError: 390018:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11482.683640] LustreError: 390003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755527 with bad export cookie 15979283911118779421 [11482.689987] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11482.697642] LustreError: 390003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11492.568362] Lustre: server umount lustre-MDT0001 complete [11503.375545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11507.965816] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11510.575215] LustreError: 394513:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11517.174739] Lustre: 394852:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.158@tcp [11518.992556] Lustre: server umount MGS complete [11530.627693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.176541] LustreError: 395577:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.199399] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11531.199409] Lustre: Skipped 1 previous similar message [11531.253609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11535.588165] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11541.472890] LustreError: 395578:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.486248] LustreError: 395578:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11544.090564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11544.301118] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11548.645720] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11555.478558] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11561.206371] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11563.250743] Lustre: Permanently reactivating lustre-OST0000 [11563.266714] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11563.270721] Lustre: Skipped 2 previous similar messages [11592.167691] LustreError: 397398:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11600.249187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11600.963925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11600.978870] Lustre: Skipped 1 previous similar message [11607.755610] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11615.413810] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11622.369626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11622.384167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.402032] Lustre: Skipped 1 previous similar message [11622.414221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11622.724823] Lustre: server umount lustre-OST0000 complete [11626.464822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11626.473835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11626.484619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11636.852416] Lustre: server umount lustre-MDT0000 complete [11638.753800] LustreError: 395574:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11638.775021] LustreError: 395574:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [11640.175344] LustreError: 395557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755685 with bad export cookie 15979283911118780471 [11640.187149] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11640.187924] LustreError: 395557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11640.517700] Lustre: server umount lustre-MDT0001 complete [11649.394868] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [11652.086213] Key type lgssc unregistered [11652.358601] LNet: 399987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11652.370201] LNetError: 399987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11652.385909] LNet: Removed LNI 192.168.204.158@tcp [11653.173874] Key type .llcrypt unregistered [11653.175808] Key type ._llcrypt unregistered [11669.509991] Key type ._llcrypt registered [11669.512881] Key type .llcrypt registered [11669.644199] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [11684.952886] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [11686.416734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11686.432235] alg: No test for adler32 (adler32-zlib) [11687.744641] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [11688.149096] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [11690.082662] Key type lgssc registered [11691.240871] Lustre: Echo OBD driver; http://www.lustre.org/ [11733.035655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11733.085914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.345428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11734.383560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11734.523755] Lustre: lustre-MDT0000: new disk, initializing [11734.639907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11734.667661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11739.515707] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11750.785641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.919980] Lustre: 403141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11751.005493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11751.009766] Lustre: Skipped 1 previous similar message [11751.135522] Lustre: lustre-MDT0001: new disk, initializing [11751.252974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11751.292555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11751.303162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11756.216802] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11766.498446] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11772.901306] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11779.837585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11780.136437] Lustre: lustre-OST0000: new disk, initializing [11780.142740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11780.150975] Lustre: 405110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11780.256644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11786.970708] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11788.874535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11788.886727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11789.012372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11795.720854] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11801.060907] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11801.321476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11806.188450] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11806.427455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11809.761881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.766736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11809.786800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11809.807193] Lustre: Skipped 1 previous similar message [11814.840768] Lustre: server umount lustre-OST0000 complete [11820.004503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11820.014296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11820.023434] Lustre: Skipped 2 previous similar messages [11820.046165] Lustre: Skipped 1 previous similar message [11825.125585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11825.140884] Lustre: Skipped 1 previous similar message [11825.871651] Lustre: server umount lustre-MDT0000 complete [11829.522079] LustreError: 403133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755874 with bad export cookie 7381061457350766414 [11829.528949] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11829.536814] LustreError: 403133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11829.907738] Lustre: server umount lustre-MDT0001 complete [11839.189086] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11840.964932] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:38:03 (1788755883) [11848.537333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11849.141205] LustreError: 407570:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11849.166356] LustreError: 407570:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11849.256488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11854.146612] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11854.304267] LustreError: 407571:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.450962] LustreError: 407570:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.040928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.377456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11869.240708] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11877.727420] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11883.667792] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11889.680978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.026882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11896.469973] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11904.310937] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11906.140185] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11908.830442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11914.720554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11914.725285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.738498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11914.754028] Lustre: Skipped 1 previous similar message [11914.769869] Lustre: Skipped 1 previous similar message [11919.852444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11919.870388] Lustre: Skipped 1 previous similar message [11920.141136] Lustre: server umount lustre-OST0000 complete [11924.960714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.082158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.089100] Lustre: Skipped 3 previous similar messages [11930.810682] Lustre: server umount lustre-MDT0000 complete [11934.670633] LustreError: 407551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755979 with bad export cookie 7381061457350767296 [11934.677547] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11934.689599] LustreError: 407551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11935.109391] Lustre: server umount lustre-MDT0001 complete [11947.043466] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [11949.833296] Key type lgssc unregistered [11950.178934] LNet: 411294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11950.183459] LNetError: 411294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11950.194164] LNet: Removed LNI 192.168.204.158@tcp [11951.152367] Key type .llcrypt unregistered [11951.156800] Key type ._llcrypt unregistered [11980.765080] Key type ._llcrypt registered [11980.767460] Key type .llcrypt registered [11980.880205] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [11981.358764] Key type .llcrypt unregistered [11981.365912] Key type ._llcrypt unregistered [12002.730403] Key type ._llcrypt registered [12002.732316] Key type .llcrypt registered [12002.834746] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12003.515660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12003.637769] alg: No test for adler32 (adler32-zlib) [12004.771858] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [12004.989388] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12006.680353] Key type lgssc registered [12008.034744] Lustre: Echo OBD driver; http://www.lustre.org/ [12016.046592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12016.066279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.678404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12021.943534] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12027.879962] LustreError: 413504:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.899507] LustreError: 413504:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12031.247813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12031.563487] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.594989] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12031.606498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12036.580903] LustreError: 413504:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.627019] LustreError: 413504:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12039.854437] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12041.706925] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12041.734973] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12042.759894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12047.844552] LustreError: 413504:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.864056] LustreError: 413504:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [12049.509401] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12058.087659] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.121610] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12058.470093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12059.046345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12064.290655] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12073.440095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12073.446330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.460336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12074.466830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12074.471272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12074.487195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12074.505848] Lustre: Skipped 1 previous similar message [12075.510548] Lustre: server umount lustre-OST0000 complete [12079.584936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12079.615115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12079.627130] Lustre: Skipped 1 previous similar message [12081.723651] Lustre: server umount lustre-MDT0000 complete [12084.705107] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12084.723989] LustreError: 413503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12086.084664] LustreError: 413486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756130 with bad export cookie 2478020256539087605 [12086.095964] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12086.102890] LustreError: 413486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12086.556511] Lustre: server umount lustre-MDT0001 complete [12097.342764] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12100.595650] Key type lgssc unregistered [12100.904478] LNet: 416654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12100.930443] LNetError: 416654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12100.953178] LNet: Removed LNI 192.168.204.158@tcp [12101.786926] Key type .llcrypt unregistered [12101.790202] Key type ._llcrypt unregistered [12127.400289] Key type ._llcrypt registered [12127.402375] Key type .llcrypt registered [12127.484792] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12128.547555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12128.581243] alg: No test for adler32 (adler32-zlib) [12129.603238] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [12129.919061] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12131.735213] Key type lgssc registered [12132.966054] Lustre: Echo OBD driver; http://www.lustre.org/ [12140.206744] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12140.229727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12155.743959] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12160.864315] LustreError: 418040:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.158@tcp: failed processing log, type 4: rc = -110 [12187.615232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12194.202716] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12201.981376] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12209.082845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12209.863632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12209.955192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12214.833609] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12215.275947] LustreError: 418065:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12215.297385] LustreError: 418065:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12220.400155] LustreError: 418066:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.435826] LustreError: 418065:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.449953] LustreError: 418065:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12224.091674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12224.362589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12228.996489] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12237.358518] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12242.948889] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12251.106785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12251.114981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12251.132758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12252.469978] Lustre: server umount lustre-OST0000 complete [12257.767763] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12257.779715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.795489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12258.449286] Lustre: server umount lustre-MDT0000 complete [12260.320601] LustreError: 419082:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.324826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12262.329364] LustreError: 418046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756307 with bad export cookie 16765871757006689642 [12262.351143] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12272.832108] Lustre: server umount lustre-MDT0001 complete [12285.267202] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12288.254763] Key type lgssc unregistered [12288.614865] LNet: 421604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12288.624078] LNetError: 421604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12288.636539] LNet: Removed LNI 192.168.204.158@tcp [12289.777304] Key type .llcrypt unregistered [12289.780966] Key type ._llcrypt unregistered [12318.770820] Key type ._llcrypt registered [12318.773133] Key type .llcrypt registered [12318.970394] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12320.210196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12320.271391] alg: No test for adler32 (adler32-zlib) [12321.316155] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [12321.556987] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12323.231375] Key type lgssc registered [12324.809517] Lustre: Echo OBD driver; http://www.lustre.org/ [12332.743455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12332.784244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12334.408056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12339.444325] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12344.802219] LustreError: 423023:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.830480] LustreError: 423023:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12349.218767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12349.754745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12355.352452] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12364.268954] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12369.838627] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12377.068970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12377.467653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12379.524408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12384.433198] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12392.740349] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12399.514110] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [12400.098321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12400.102096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12400.107573] Lustre: Skipped 1 previous similar message [12400.121175] Lustre: Skipped 1 previous similar message [12402.791451] Lustre: server umount lustre-MDT0000 complete [12404.600368] LustreError: 423018:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.144605] LustreError: 423004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.58@tcp arrived at 1788756449 with bad export cookie 5482404915801793206 [12405.168068] LustreError: 423004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12407.251965] LustreError: 423003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756452 with bad export cookie 5482404915801792317 [12407.265319] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12407.268709] LustreError: 423003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12410.337949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12410.338776] LustreError: 423018:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.342291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12410.342299] Lustre: Skipped 1 previous similar message [12410.343035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12410.361035] LustreError: 423018:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [12412.384378] LustreError: 423023:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.384623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.409630] LustreError: 423023:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12413.497965] Lustre: server umount lustre-OST0000 complete [12417.505568] LustreError: 423019:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.525873] LustreError: 423019:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12422.192423] Lustre: server umount lustre-MDT0001 complete [12436.490479] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12439.751462] Key type lgssc unregistered [12440.210348] LNet: 426710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12440.227627] LNetError: 426710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12441.259992] LNet: Removed LNI 192.168.204.158@tcp [12442.495108] Key type .llcrypt unregistered [12442.496960] Key type ._llcrypt unregistered [12476.462747] Key type ._llcrypt registered [12476.465324] Key type .llcrypt registered [12476.689309] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12478.025956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12478.299522] alg: No test for adler32 (adler32-zlib) [12479.550326] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [12479.872484] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12481.647304] Key type lgssc registered [12483.214898] Lustre: Echo OBD driver; http://www.lustre.org/ [12492.020951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12492.041191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12493.679251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12499.179697] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12504.038143] LustreError: 428127:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12504.055887] LustreError: 428127:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12507.568106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.185140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12513.062739] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12521.071499] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12526.975055] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12534.235035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12534.684455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12538.808746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12541.926734] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12550.020088] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12558.202924] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (stopping) [12558.309179] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.398256] Lustre: server umount lustre-MDT0001 complete [12563.331923] LustreError: 428123:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12568.546089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.546341] LustreError: 428126:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12568.559698] Lustre: Skipped 1 previous similar message [12568.560658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12568.560662] Lustre: Skipped 2 previous similar messages [12568.604996] LustreError: 428126:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [12573.668432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12573.668655] LustreError: 428127:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.610914] Lustre: server umount lustre-OST0000 complete [12578.625607] Lustre: server umount lustre-MDT0000 complete [12594.663431] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12597.676973] Key type lgssc unregistered [12598.019705] LNet: 431808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12598.032648] LNetError: 431808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12598.065762] LNet: Removed LNI 192.168.204.158@tcp [12598.968871] Key type .llcrypt unregistered [12598.971312] Key type ._llcrypt unregistered [12625.633511] Key type ._llcrypt registered [12625.635439] Key type .llcrypt registered [12625.741856] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [12644.072314] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12646.522736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12646.594389] alg: No test for adler32 (adler32-zlib) [12647.713036] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [12648.130088] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12649.887969] Key type lgssc registered [12651.135129] Lustre: Echo OBD driver; http://www.lustre.org/ [12695.774973] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12712.130925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12712.183856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.505582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12713.561405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12713.734255] Lustre: lustre-MDT0000: new disk, initializing [12713.865448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12713.885987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12718.539771] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12733.992747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12734.275731] Lustre: lustre-OST0000: new disk, initializing [12734.278592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12734.284600] Lustre: Skipped 1 previous similar message [12734.289697] Lustre: 436843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12734.413619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12741.534723] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12741.730076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12741.735222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12741.760293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12751.837672] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12770.098243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12770.270035] Lustre: 435825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12770.341724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12770.431324] Lustre: lustre-MDT0001: new disk, initializing [12770.587173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12770.686475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12770.705512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12775.457686] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12784.815052] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12785.048430] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12790.477634] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12790.711909] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12799.466381] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12799.467341] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12799.491875] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12799.516284] Lustre: Skipped 1 previous similar message [12800.422373] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (stopping) [12804.580110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12804.595035] Lustre: Skipped 1 previous similar message [12809.703700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12809.717622] Lustre: Skipped 2 previous similar messages [12810.210851] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12810.538827] Lustre: server umount lustre-MDT0001 complete [12815.733945] LustreError: 435830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.769720] LustreError: 435830:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [12819.939778] LustreError: 436851:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12819.948968] LustreError: 436851:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12825.067525] LustreError: 435831:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12825.117187] LustreError: 435831:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12830.185550] LustreError: 435830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12830.209196] LustreError: 435830:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12833.761255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12833.772111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12833.788081] Lustre: Skipped 1 previous similar message [12833.800747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12835.305531] LustreError: 435835:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12836.825294] Lustre: server umount lustre-OST0000 complete [12849.030852] Lustre: server umount lustre-MDT0000 complete [12863.800360] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12866.547624] Key type lgssc unregistered [12866.949589] LNet: 440593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12866.963112] LNetError: 440593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12866.979558] LNet: Removed LNI 192.168.204.158@tcp [12867.978071] Key type .llcrypt unregistered [12867.982454] Key type ._llcrypt unregistered [12911.797520] Key type ._llcrypt registered [12911.799671] Key type .llcrypt registered [12911.901762] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12912.181504] Key type .llcrypt unregistered [12912.190587] Key type ._llcrypt unregistered [12931.227395] Key type ._llcrypt registered [12931.237135] Key type .llcrypt registered [12931.413787] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12932.846178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12933.040129] alg: No test for adler32 (adler32-zlib) [12934.488475] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [12935.109641] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12936.952439] Key type lgssc registered [12939.012648] Lustre: Echo OBD driver; http://www.lustre.org/ [12977.835719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12977.908371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12979.347753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12979.403907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12979.603948] Lustre: lustre-MDT0000: new disk, initializing [12979.794909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12979.832290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12986.322552] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13002.852187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.003120] Lustre: 443964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13003.086618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13003.090821] Lustre: Skipped 1 previous similar message [13003.221311] Lustre: lustre-MDT0001: new disk, initializing [13003.350809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13003.434115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13003.452456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13009.734250] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13022.998767] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13031.278556] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13039.437594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13039.741613] Lustre: lustre-OST0000: new disk, initializing [13039.747444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13039.757840] Lustre: 445934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13039.878411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13046.422110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13046.436719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13046.644164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13049.198110] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13061.296316] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13069.796085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13069.804082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13069.818321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13072.362620] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13072.367154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13072.380747] Lustre: Skipped 1 previous similar message [13072.402200] Lustre: Skipped 2 previous similar messages [13074.839425] Lustre: server umount lustre-MDT0000 complete [13079.675120] LustreError: 443955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757124 with bad export cookie 8780910013195170236 [13079.679390] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13079.685937] LustreError: 443955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13080.209806] Lustre: server umount lustre-MDT0001 complete [13095.009620] Lustre: server umount lustre-OST0000 complete [13107.072087] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:59:09 (1788757149) [13116.075718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13116.544216] LustreError: 447965:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13116.582720] LustreError: 447965:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [13116.690186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13122.017471] LustreError: 447966:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13122.866754] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13127.137290] LustreError: 447965:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.258319] LustreError: 447966:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13134.036484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13134.611820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13140.850835] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13150.550524] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13158.654802] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13180.402543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13180.762995] Lustre: Found index 0 for lustre-OST0000, updating log [13181.106318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13186.228745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13188.198739] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13197.210663] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13204.738114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13204.752678] LustreError: 449794:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13204.907942] Lustre: server umount lustre-OST0000 complete [13226.468410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13226.479444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13226.490802] Lustre: Skipped 1 previous similar message [13228.326459] Lustre: server umount lustre-MDT0000 complete [13231.587086] LustreError: 448906:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13231.602104] LustreError: 448906:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13233.212479] LustreError: 447946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757278 with bad export cookie 8780910013195171629 [13233.219769] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13233.232976] LustreError: 447946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13233.634440] Lustre: server umount lustre-MDT0001 complete [13243.250975] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:01:25 (1788757285) [13250.527700] LNet: 1 local NIs in recovery (showing 1): 192.168.204.158@tcp [13869.011508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13869.465487] LustreError: 454389:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13869.558450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13873.768361] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13874.656366] LustreError: 454390:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13879.789039] LustreError: 454389:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13882.563790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13883.036056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13887.142723] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13894.541992] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13899.623376] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13918.896168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13919.213262] Lustre: Found index 0 for lustre-OST0000, updating log [13919.556194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13925.817304] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13927.392111] LNet: 1 local NIs in recovery (showing 1): 192.168.204.158@tcp [13933.244737] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13935.449267] Lustre: server umount lustre-OST0000 complete [13940.582907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13940.597814] Lustre: Skipped 1 previous similar message [13940.603189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13940.611430] Lustre: Skipped 1 previous similar message [13941.666439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.674370] Lustre: Skipped 1 previous similar message [13942.691804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13942.698865] Lustre: Skipped 1 previous similar message [13944.739043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13944.751790] Lustre: Skipped 3 previous similar messages [13945.766982] LustreError: 454385:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13945.797121] LustreError: 454385:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13945.985480] Lustre: server umount lustre-MDT0000 complete [13950.082132] LustreError: 454371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757994 with bad export cookie 8780910013195172700 [13950.093972] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13950.097028] LustreError: 454371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13950.494671] Lustre: server umount lustre-MDT0001 complete [13980.041486] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [13990.779295] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14026.441600] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:14:29 (1788758069) [14045.458764] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [14048.592823] Key type lgssc unregistered [14048.893144] LNet: 461009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14048.897686] LNetError: 461009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14173.879497] LNet: Removed LNI 192.168.204.158@tcp [14174.674901] Key type .llcrypt unregistered [14174.677161] Key type ._llcrypt unregistered [14191.989245] Key type ._llcrypt registered [14191.992255] Key type .llcrypt registered [14192.095083] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14193.195257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14193.371099] alg: No test for adler32 (adler32-zlib) [14194.452393] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [14194.659219] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [14196.399441] Key type lgssc registered [14197.689804] Lustre: Echo OBD driver; http://www.lustre.org/ [14202.317316] LNet: Added LNI 192.168.204.158@tcp100 [8/256/0/180] [14203.331054] LNet: Added LNI 192.168.204.158@tcp101 [8/256/0/180] [14204.256281] LNet: Added LNI 192.168.204.158@tcp102 [8/256/0/180] [14205.105045] LNet: Added LNI 192.168.204.158@tcp103 [8/256/0/180] [14205.940341] LNet: Added LNI 192.168.204.158@tcp104 [8/256/0/180] [14206.985644] LNet: Added LNI 192.168.204.158@tcp105 [8/256/0/180] [14207.871969] LNet: Added LNI 192.168.204.158@tcp106 [8/256/0/180] [14208.818384] LNet: Added LNI 192.168.204.158@tcp107 [8/256/0/180] [14209.736431] LNet: Added LNI 192.168.204.158@tcp108 [8/256/0/180] [14210.589309] LNet: Added LNI 192.168.204.158@tcp109 [8/256/0/180] [14211.561143] LNet: Added LNI 192.168.204.158@tcp110 [8/256/0/180] [14212.469735] LNet: Added LNI 192.168.204.158@tcp111 [8/256/0/180] [14213.263873] LNet: Added LNI 192.168.204.158@tcp112 [8/256/0/180] [14214.039936] LNet: Added LNI 192.168.204.158@tcp113 [8/256/0/180] [14214.768981] LNet: Added LNI 192.168.204.158@tcp114 [8/256/0/180] [14215.481753] LNet: Added LNI 192.168.204.158@tcp115 [8/256/0/180] [14216.237290] LNet: Added LNI 192.168.204.158@tcp116 [8/256/0/180] [14217.027320] LNet: Added LNI 192.168.204.158@tcp117 [8/256/0/180] [14218.054355] LNet: Added LNI 192.168.204.158@tcp118 [8/256/0/180] [14219.155260] LNet: Added LNI 192.168.204.158@tcp119 [8/256/0/180] [14220.203505] LNet: Added LNI 192.168.204.158@tcp120 [8/256/0/180] [14221.279964] LNet: Added LNI 192.168.204.158@tcp121 [8/256/0/180] [14222.263232] LNet: Added LNI 192.168.204.158@tcp122 [8/256/0/180] [14223.184932] LNet: Added LNI 192.168.204.158@tcp123 [8/256/0/180] [14224.189710] LNet: Added LNI 192.168.204.158@tcp124 [8/256/0/180] [14225.174928] LNet: Added LNI 192.168.204.158@tcp125 [8/256/0/180] [14226.100826] LNet: Added LNI 192.168.204.158@tcp126 [8/256/0/180] [14226.914917] LNet: Added LNI 192.168.204.158@tcp127 [8/256/0/180] [14228.026294] LNet: Added LNI 192.168.204.158@tcp128 [8/256/0/180] [14229.055734] LNet: Added LNI 192.168.204.158@tcp129 [8/256/0/180] [14230.153580] LNet: Added LNI 192.168.204.158@tcp130 [8/256/0/180] [14231.245523] LNet: Added LNI 192.168.204.158@tcp131 [8/256/0/180] [14232.152026] LNet: Added LNI 192.168.204.158@tcp132 [8/256/0/180] [14233.179819] LNet: Added LNI 192.168.204.158@tcp133 [8/256/0/180] [14234.330748] LNet: Added LNI 192.168.204.158@tcp134 [8/256/0/180] [14235.334137] LNet: Added LNI 192.168.204.158@tcp135 [8/256/0/180] [14243.096678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14243.114205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14244.299213] Lustre: 464040:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [14244.388969] LNetError: 461497:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14244.421395] LNetError: Protocol error connecting to 192.168.204.158@tcp116 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14244.438019] LNetError: Skipped 1 previous similar message [14244.580715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14244.627315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14244.735953] Lustre: lustre-MDT0000: new disk, initializing [14244.847510] Lustre: 464096:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [14244.867266] Lustre: 464096:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14244.896783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14244.917930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14245.474877] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14245.491029] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14245.513169] LNetError: Protocol error connecting to 192.168.204.158@tcp117 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14245.537346] LNetError: Skipped 2 previous similar messages [14250.563307] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14262.743357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14262.980514] Lustre: 464056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14263.059803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14263.064994] Lustre: Skipped 1 previous similar message [14263.152248] Lustre: lustre-MDT0001: new disk, initializing [14263.195384] Lustre: 464859:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [14263.205582] Lustre: 464859:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14263.341125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14263.402321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14263.430795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14269.173653] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14273.093900] Lustre: 465244:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [14273.105304] Lustre: 465244:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14274.532333] LNetError: 464061:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14274.552830] LNetError: Protocol error connecting to 192.168.204.158@tcp117 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14276.582368] LNetError: 464063:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14276.605554] LNetError: Protocol error connecting to 192.168.204.158@tcp120 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14279.882731] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14281.701176] LNetError: 461498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14281.715403] LNetError: 461498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14281.724491] LNetError: Protocol error connecting to 192.168.204.158@tcp121 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14281.747449] LNetError: Skipped 1 previous similar message [14286.161443] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14289.890867] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14289.900510] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14289.909630] LNetError: Protocol error connecting to 192.168.204.158@tcp120 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14289.919377] LNetError: Skipped 4 previous similar messages [14293.311370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14293.663639] Lustre: lustre-OST0000: new disk, initializing [14293.669631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14293.678742] Lustre: 466032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14293.785153] Lustre: 466037:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [14293.791038] Lustre: 466037:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14293.844599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14300.456784] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14301.262643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14301.270511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14301.398963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14306.274186] LNetError: 461497:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14306.292352] LNetError: 461497:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14306.302130] LNetError: Protocol error connecting to 192.168.204.158@tcp121 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14306.310025] LNetError: Skipped 8 previous similar messages [14311.217500] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14322.147107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14322.149451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14322.152144] LustreError: Skipped 1 previous similar message [14322.153843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14322.164816] Lustre: Skipped 1 previous similar message [14322.190457] Lustre: Skipped 1 previous similar message [14326.410417] Lustre: server umount lustre-OST0000 complete [14332.385735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14332.398939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14332.409672] Lustre: Skipped 1 previous similar message [14332.440031] Lustre: Skipped 1 previous similar message [14337.837771] Lustre: server umount lustre-MDT0000 complete [14340.066535] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [14340.078670] LNetError: 464067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14340.091512] LNetError: Protocol error connecting to 192.168.204.158@tcp120 at host 192.168.204.158:988: is it running a compatible version of Lustre? [14340.109600] LNetError: Skipped 10 previous similar messages [14342.601558] LustreError: 464048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758387 with bad export cookie 13265978056837283346 [14342.617274] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14342.621026] LustreError: 464048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14343.088595] Lustre: server umount lustre-MDT0001 complete [14356.414236] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [14359.445542] Key type lgssc unregistered [14359.827297] LNet: 467945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14359.840888] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14359.846778] LNet: Removed LNI 192.168.204.158@tcp [14360.863374] LNet: Removed LNI 192.168.204.158@tcp100 [14360.869951] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14360.882608] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14360.890205] LNet: Removed LNI 192.168.204.158@tcp101 [14361.951297] LNet: Removed LNI 192.168.204.158@tcp102 [14361.963931] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14361.976948] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14361.987392] LNet: Removed LNI 192.168.204.158@tcp103 [14363.039736] LNet: Removed LNI 192.168.204.158@tcp104 [14364.068397] LNet: Removed LNI 192.168.204.158@tcp105 [14364.072174] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14364.080691] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14364.085135] LNet: Removed LNI 192.168.204.158@tcp106 [14365.151686] LNet: Removed LNI 192.168.204.158@tcp107 [14366.175591] LNet: Removed LNI 192.168.204.158@tcp108 [14367.199184] LNet: Removed LNI 192.168.204.158@tcp109 [14368.223309] LNet: Removed LNI 192.168.204.158@tcp110 [14368.231648] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14368.239608] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14368.256241] LNet: Removed LNI 192.168.204.158@tcp111 [14369.311726] LNet: Removed LNI 192.168.204.158@tcp112 [14370.336467] LNet: Removed LNI 192.168.204.158@tcp113 [14371.359159] LNet: Removed LNI 192.168.204.158@tcp114 [14372.383498] LNet: Removed LNI 192.168.204.158@tcp115 [14373.407175] LNet: Removed LNI 192.168.204.158@tcp116 [14374.431371] LNet: Removed LNI 192.168.204.158@tcp117 [14375.456409] LNet: Removed LNI 192.168.204.158@tcp118 [14376.479527] LNet: Removed LNI 192.168.204.158@tcp119 [14376.483045] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14376.488753] LNetError: 467945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14376.502951] LNet: Removed LNI 192.168.204.158@tcp120 [14377.571817] LNet: Removed LNI 192.168.204.158@tcp121 [14378.591196] LNet: Removed LNI 192.168.204.158@tcp122 [14378.597134] LNet: Removed LNI 192.168.204.158@tcp123 [14378.601224] LNet: Removed LNI 192.168.204.158@tcp124 [14378.607800] LNet: Removed LNI 192.168.204.158@tcp125 [14378.614140] LNet: Removed LNI 192.168.204.158@tcp126 [14378.621886] LNet: Removed LNI 192.168.204.158@tcp127 [14378.624772] LNet: Removed LNI 192.168.204.158@tcp128 [14378.626873] LNet: Removed LNI 192.168.204.158@tcp129 [14378.630406] LNet: Removed LNI 192.168.204.158@tcp130 [14378.632747] LNet: Removed LNI 192.168.204.158@tcp131 [14378.636450] LNet: Removed LNI 192.168.204.158@tcp132 [14378.639498] LNet: Removed LNI 192.168.204.158@tcp133 [14378.642483] LNet: Removed LNI 192.168.204.158@tcp134 [14378.654292] LNet: Removed LNI 192.168.204.158@tcp135 [14379.486345] Key type .llcrypt unregistered [14379.489057] Key type ._llcrypt unregistered [14402.472992] Key type ._llcrypt registered [14402.489483] Key type .llcrypt registered [14402.643620] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [14418.559260] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14420.031761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14420.061690] alg: No test for adler32 (adler32-zlib) [14421.137872] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [14421.364874] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [14423.065655] Key type lgssc registered [14424.347932] Lustre: Echo OBD driver; http://www.lustre.org/ [14466.603728] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:21:49 (1788758509) [14485.510581] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [14488.038263] Key type lgssc unregistered [14488.365418] LNet: 472027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14488.376418] LNetError: 472027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14488.388473] LNet: Removed LNI 192.168.204.158@tcp [14489.414857] Key type .llcrypt unregistered [14489.418815] Key type ._llcrypt unregistered [14509.393690] Key type ._llcrypt registered [14509.401962] Key type .llcrypt registered [14509.610831] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14511.470256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14511.713043] alg: No test for adler32 (adler32-zlib) [14512.917944] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [14513.162113] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [14514.927291] Key type lgssc registered [14517.505776] Lustre: Echo OBD driver; http://www.lustre.org/ [14529.197549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14529.219509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14530.590491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14530.679273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14530.759610] Lustre: lustre-MDT0000: new disk, initializing [14530.887702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14530.917404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14535.763923] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14546.865829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14546.994691] Lustre: 473236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14547.116534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14547.123366] Lustre: Skipped 1 previous similar message [14547.252563] Lustre: lustre-MDT0001: new disk, initializing [14547.385152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14547.422752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14547.433789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14552.860020] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14563.040590] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14568.997647] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14576.203810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14576.564873] Lustre: lustre-OST0000: new disk, initializing [14576.572873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14576.578789] Lustre: 475205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14576.676891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14583.177354] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14586.509809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14586.533695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14586.695737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14592.568614] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14596.806470] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing lnet_if_list [14602.699974] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14602.733410] LNet: Added LNI 192.168.204.158@tcp10 [8/256/0/180] [14608.980710] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14609.039601] LNet: Added LNI 192.168.204.158@tcp11 [8/256/0/180] [14613.097173] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14613.148905] LNet: Added LNI 192.168.204.158@tcp12 [8/256/0/180] [14617.948741] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14617.973984] LNet: Added LNI 192.168.204.158@tcp13 [8/256/0/180] [14622.497156] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14622.550866] LNet: Added LNI 192.168.204.158@tcp14 [8/256/0/180] [14628.016331] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14628.060501] LNet: Added LNI 192.168.204.158@tcp15 [8/256/0/180] [14633.930442] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14633.999446] LNet: Added LNI 192.168.204.158@tcp16 [8/256/0/180] [14639.891081] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14639.939814] LNet: Added LNI 192.168.204.158@tcp17 [8/256/0/180] [14647.250221] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14647.294502] LNet: Added LNI 192.168.204.158@tcp18 [8/256/0/180] [14653.912766] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14653.952384] LNet: Added LNI 192.168.204.158@tcp19 [8/256/0/180] [14660.501622] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14660.550720] LNet: Added LNI 192.168.204.158@tcp20 [8/256/0/180] [14667.022959] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14667.075158] LNet: Added LNI 192.168.204.158@tcp21 [8/256/0/180] [14673.203484] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14673.231730] LNet: Added LNI 192.168.204.158@tcp22 [8/256/0/180] [14679.665310] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14679.698205] LNet: Added LNI 192.168.204.158@tcp23 [8/256/0/180] [14685.748782] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14685.773959] LNet: Added LNI 192.168.204.158@tcp24 [8/256/0/180] [14691.686160] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14691.749352] LNet: Added LNI 192.168.204.158@tcp25 [8/256/0/180] [14699.422296] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14699.474398] LNet: Added LNI 192.168.204.158@tcp26 [8/256/0/180] [14708.421717] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14708.493437] LNet: Added LNI 192.168.204.158@tcp27 [8/256/0/180] [14717.407928] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14717.458770] LNet: Added LNI 192.168.204.158@tcp28 [8/256/0/180] [14726.235233] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14726.268178] LNet: Added LNI 192.168.204.158@tcp29 [8/256/0/180] [14733.124734] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14733.149753] LNet: Added LNI 192.168.204.158@tcp30 [8/256/0/180] [14740.024666] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14740.056251] LNet: Added LNI 192.168.204.158@tcp31 [8/256/0/180] [14746.643286] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14746.678235] LNet: Added LNI 192.168.204.158@tcp32 [8/256/0/180] [14753.473329] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14753.499572] LNet: Added LNI 192.168.204.158@tcp33 [8/256/0/180] [14761.011263] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14761.038886] LNet: Added LNI 192.168.204.158@tcp34 [8/256/0/180] [14768.412578] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14768.462186] LNet: Added LNI 192.168.204.158@tcp35 [8/256/0/180] [14775.939874] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14775.965048] LNet: Added LNI 192.168.204.158@tcp36 [8/256/0/180] [14783.325979] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14783.385818] LNet: Added LNI 192.168.204.158@tcp37 [8/256/0/180] [14791.674695] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14791.702814] LNet: Added LNI 192.168.204.158@tcp38 [8/256/0/180] [14800.006211] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14800.033906] LNet: Added LNI 192.168.204.158@tcp39 [8/256/0/180] [14808.336656] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14808.367156] LNet: Added LNI 192.168.204.158@tcp40 [8/256/0/180] [14817.652665] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14817.708816] LNet: Added LNI 192.168.204.158@tcp41 [8/256/0/180] [14825.807563] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14825.836079] LNet: Added LNI 192.168.204.158@tcp42 [8/256/0/180] [14833.700857] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14833.740502] LNet: Added LNI 192.168.204.158@tcp43 [8/256/0/180] [14842.082284] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14842.126764] LNet: Added LNI 192.168.204.158@tcp44 [8/256/0/180] [14850.306880] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14850.343488] LNet: Added LNI 192.168.204.158@tcp45 [8/256/0/180] [14858.304205] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14858.348045] LNet: Added LNI 192.168.204.158@tcp46 [8/256/0/180] [14867.316198] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14867.353185] LNet: Added LNI 192.168.204.158@tcp47 [8/256/0/180] [14876.656328] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14876.692213] LNet: Added LNI 192.168.204.158@tcp48 [8/256/0/180] [14885.829957] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14885.879754] LNet: Added LNI 192.168.204.158@tcp49 [8/256/0/180] [14895.927708] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14895.999585] LNet: Added LNI 192.168.204.158@tcp50 [8/256/0/180] [14914.534279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14914.550995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14914.551451] Lustre: Skipped 2 previous similar messages [14914.556842] Lustre: Skipped 1 previous similar message [14919.652032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14919.658631] Lustre: Skipped 3 previous similar messages [14920.700625] Lustre: server umount lustre-MDT0000 complete [14924.562949] LustreError: 474031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758969 with bad export cookie 6026525920619514788 [14924.571743] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14924.580742] LustreError: 474031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14924.769187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14924.787707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14925.004181] Lustre: server umount lustre-MDT0001 complete [14942.903222] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14951.370488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14970.125030] LustreError: 475199:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14970.142541] LustreError: 475199:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [14970.194740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14975.067399] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14975.458883] LustreError: 475200:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14979.559581] LustreError: 475199:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14979.566401] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14984.258739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14984.600134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14989.758550] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14990.106120] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14998.718399] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15006.374895] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15030.769493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15030.788083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15030.788492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15030.806982] Lustre: Skipped 1 previous similar message [15035.441383] Lustre: server umount lustre-OST0000 complete [15040.993989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15040.994257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15041.003370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15041.026026] Lustre: Skipped 2 previous similar messages [15046.855497] Lustre: server umount lustre-MDT0000 complete [15050.211847] LustreError: 499453:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15050.239073] LustreError: 499453:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15050.961244] LustreError: 473228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759095 with bad export cookie 6026525920619517434 [15050.971012] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15050.980878] LustreError: 473228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15051.340979] Lustre: server umount lustre-MDT0001 complete [15064.803571] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [15068.686100] Key type lgssc unregistered [15069.324864] LNet: 502250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15069.341326] LNetError: 502250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15069.362516] LNet: Removed LNI 192.168.204.158@tcp [15069.369712] LNet: Removed LNI 192.168.204.158@tcp10 [15069.378413] LNet: Removed LNI 192.168.204.158@tcp11 [15069.381696] LNet: Removed LNI 192.168.204.158@tcp12 [15069.388666] LNet: Removed LNI 192.168.204.158@tcp13 [15069.398914] LNet: Removed LNI 192.168.204.158@tcp14 [15069.408815] LNet: Removed LNI 192.168.204.158@tcp15 [15069.410921] LNet: Removed LNI 192.168.204.158@tcp16 [15069.423529] LNet: Removed LNI 192.168.204.158@tcp17 [15069.427809] LNet: Removed LNI 192.168.204.158@tcp18 [15069.443833] LNet: Removed LNI 192.168.204.158@tcp19 [15069.447724] LNet: Removed LNI 192.168.204.158@tcp20 [15069.461732] LNet: Removed LNI 192.168.204.158@tcp21 [15069.472342] LNet: Removed LNI 192.168.204.158@tcp22 [15069.480687] LNet: Removed LNI 192.168.204.158@tcp23 [15069.492082] LNet: Removed LNI 192.168.204.158@tcp24 [15069.498786] LNet: Removed LNI 192.168.204.158@tcp25 [15069.500423] LNet: Removed LNI 192.168.204.158@tcp26 [15069.503636] LNet: Removed LNI 192.168.204.158@tcp27 [15069.508377] LNet: Removed LNI 192.168.204.158@tcp28 [15069.512649] LNet: Removed LNI 192.168.204.158@tcp29 [15069.515499] LNet: Removed LNI 192.168.204.158@tcp30 [15069.518560] LNet: Removed LNI 192.168.204.158@tcp31 [15069.523386] LNet: Removed LNI 192.168.204.158@tcp32 [15069.527510] LNet: Removed LNI 192.168.204.158@tcp33 [15069.529285] LNet: Removed LNI 192.168.204.158@tcp34 [15069.539258] LNet: Removed LNI 192.168.204.158@tcp35 [15069.544283] LNet: Removed LNI 192.168.204.158@tcp36 [15069.547337] LNet: Removed LNI 192.168.204.158@tcp37 [15069.550065] LNet: Removed LNI 192.168.204.158@tcp38 [15069.554074] LNet: Removed LNI 192.168.204.158@tcp39 [15069.557523] LNet: Removed LNI 192.168.204.158@tcp40 [15069.560475] LNet: Removed LNI 192.168.204.158@tcp41 [15069.564064] LNet: Removed LNI 192.168.204.158@tcp42 [15069.570145] LNet: Removed LNI 192.168.204.158@tcp43 [15069.573163] LNet: Removed LNI 192.168.204.158@tcp44 [15069.576404] LNet: Removed LNI 192.168.204.158@tcp45 [15069.582596] LNet: Removed LNI 192.168.204.158@tcp46 [15069.588703] LNet: Removed LNI 192.168.204.158@tcp47 [15069.598325] LNet: Removed LNI 192.168.204.158@tcp48 [15069.603894] LNet: Removed LNI 192.168.204.158@tcp49 [15069.626465] LNet: Removed LNI 192.168.204.158@tcp50 [15070.785324] Key type .llcrypt unregistered [15070.787166] Key type ._llcrypt unregistered [15094.960545] Key type ._llcrypt registered [15094.963448] Key type .llcrypt registered [15095.094676] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [15111.194638] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [15112.127103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15112.173248] alg: No test for adler32 (adler32-zlib) [15113.392469] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [15113.646107] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [15115.384229] Key type lgssc registered [15116.660817] Lustre: Echo OBD driver; http://www.lustre.org/ [15155.522976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15155.584354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15156.908980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15156.963717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15157.056899] Lustre: lustre-MDT0000: new disk, initializing [15157.196605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15157.226692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15164.530039] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15179.830160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15179.969463] Lustre: 505445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15180.043756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15180.050359] Lustre: Skipped 1 previous similar message [15180.221497] Lustre: lustre-MDT0001: new disk, initializing [15180.462368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15180.500155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15180.514095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15186.460476] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15197.486426] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15205.577101] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15215.277596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15215.478340] Lustre: *** cfs_fail_loc=90f, val=0*** [15215.561181] Lustre: lustre-OST0000: new disk, initializing [15215.564777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15215.570567] Lustre: 507462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15215.662628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15222.936744] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15224.389467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15224.407904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15224.542433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15234.307931] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15245.281651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15245.283738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15245.299114] LustreError: Skipped 1 previous similar message [15245.303746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15245.361958] Lustre: Skipped 1 previous similar message [15249.923106] Lustre: server umount lustre-OST0000 complete [15255.521514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15255.548880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15255.559286] Lustre: Skipped 2 previous similar messages [15260.518559] Lustre: server umount lustre-MDT0000 complete [15264.535718] LustreError: 506241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759309 with bad export cookie 9550573765050940414 [15264.546600] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15264.553055] LustreError: 506241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15265.080864] Lustre: server umount lustre-MDT0001 complete [15276.207624] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [15279.586632] Key type lgssc unregistered [15279.993448] LNet: 509371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15280.010102] LNetError: 509371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15280.025651] LNet: Removed LNI 192.168.204.158@tcp [15280.870958] Key type .llcrypt unregistered [15280.874313] Key type ._llcrypt unregistered [15302.774930] Key type ._llcrypt registered [15302.777228] Key type .llcrypt registered [15302.897563] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [15303.835233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15303.937050] alg: No test for adler32 (adler32-zlib) [15305.169415] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [15305.447368] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [15307.135237] Key type lgssc registered [15308.059467] Lustre: Echo OBD driver; http://www.lustre.org/ [15315.845922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15315.874631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15317.332627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15322.310291] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15327.719859] LustreError: 510635:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15327.749932] LustreError: 510635:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [15330.191842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15330.624081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15337.085305] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15346.285148] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15353.064659] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15360.884229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15361.396369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15369.251665] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15379.172807] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15390.999089] loop0: detected capacity change from 0 to 204800000 [15391.310428] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15391.323302] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15391.348344] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15397.993174] loop0: detected capacity change from 0 to 204800000 [15398.384092] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15398.409603] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15398.423938] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15405.170642] loop0: detected capacity change from 0 to 204800000 [15412.413653] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15412.644170] Lustre: fs15246-OST0000: new disk, initializing [15412.648348] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15412.663737] Lustre: 514231:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15412.747504] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15420.656345] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15431.483442] loop1: detected capacity change from 0 to 204800000 [15437.444517] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15437.552939] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15437.634958] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15437.724183] Lustre: fs15246-MDT0000: new disk, initializing [15437.920480] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15437.949098] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15438.073892] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15439.138687] Lustre: 515358:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000375f543d x1875650124237312/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788759499 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15443.126846] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15452.651411] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15452.660898] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15452.688265] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15468.006955] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15468.041734] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15470.426674] Lustre: server umount fs15246-MDT0000 complete [15473.135159] LustreError: 515590:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15478.241527] LustreError: 510631:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15480.056570] Lustre: server umount fs15246-OST0000 complete [15493.605722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15493.611236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15493.611518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15493.626633] Lustre: Skipped 1 previous similar message [15497.468443] Lustre: server umount lustre-OST0000 complete [15503.840875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15503.846132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15503.849682] Lustre: Skipped 1 previous similar message [15503.853360] Lustre: Skipped 1 previous similar message [15507.838646] Lustre: server umount lustre-MDT0000 complete [15508.960730] LustreError: 511559:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15508.983885] LustreError: 511559:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15511.942262] LustreError: 510615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759556 with bad export cookie 16778176739939435125 [15511.954239] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15511.957929] LustreError: 510615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15512.264929] Lustre: server umount lustre-MDT0001 complete [15524.191735] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [15527.124360] Key type lgssc unregistered [15527.416765] LNet: 518453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15527.435083] LNetError: 518453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15527.463781] LNet: Removed LNI 192.168.204.158@tcp [15528.587590] Key type .llcrypt unregistered [15528.589790] Key type ._llcrypt unregistered [15553.021268] Key type ._llcrypt registered [15553.025241] Key type .llcrypt registered [15553.157344] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [15555.272071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15555.342683] alg: No test for adler32 (adler32-zlib) [15556.692714] Lustre: Lustre: Build Version: 2.17.58_44_g07a5b2c [15557.104974] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [15558.768422] Key type lgssc registered [15560.813135] Lustre: Echo OBD driver; http://www.lustre.org/ [15611.484650] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [15623.655749] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [15663.592614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15663.639515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15665.021818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15665.090835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15665.221807] Lustre: lustre-MDT0000: new disk, initializing [15665.325346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15665.357404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15670.576703] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15682.256696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15682.449217] Lustre: 522927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15682.526819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15682.533262] Lustre: Skipped 1 previous similar message [15682.623357] Lustre: lustre-MDT0001: new disk, initializing [15682.746514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15682.786362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15682.809520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15688.478251] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15698.677122] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15704.476188] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15712.316358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15712.710479] Lustre: lustre-OST0000: new disk, initializing [15712.713489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15712.720327] Lustre: 524894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15712.827525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15719.946590] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [15722.060666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15722.075537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15722.151868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15729.159422] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15734.151816] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15734.354563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15739.008513] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15739.232907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15742.946491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15742.963605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15742.969550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15742.985162] Lustre: Skipped 1 previous similar message [15747.424237] Lustre: server umount lustre-OST0000 complete [15753.187865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15753.196134] Lustre: Skipped 1 previous similar message [15753.217034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15753.223191] Lustre: Skipped 1 previous similar message [15758.578562] Lustre: server umount lustre-MDT0000 complete [15762.821661] LustreError: 522917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759807 with bad export cookie 15654024501234121255 [15762.825093] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15762.831799] LustreError: 522917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15763.177714] Lustre: server umount lustre-MDT0001 complete [15790.709906] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [15803.109272] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [15839.766653] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15516 sec ========= 01:44:42 (1788759882) [15841.712747] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:44:44 (1788759884) === [15845.356838] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:44:48 (1788759888) === [15894.866424] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [15898.179708] Key type lgssc unregistered [15898.586621] LNet: 531542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15898.597205] LNetError: 531542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15898.624280] LNet: Removed LNI 192.168.204.158@tcp [15899.932028] Key type .llcrypt unregistered [15899.935078] Key type ._llcrypt unregistered