[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499465548 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2496MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.008341] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010004] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011016] pid_max: default: 32768 minimum: 301 [ 0.012138] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014035] SELinux: Initializing. [ 0.015083] *** VALIDATE selinux *** [ 0.023668] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027801] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030128] *** VALIDATE tmpfs *** [ 0.031510] *** VALIDATE proc *** [ 0.032324] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040934] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.042178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045016] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051308] rcu: Hierarchical SRCU implementation. [ 0.053372] smp: Bringing up secondary CPUs ... [ 0.054517] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.058066] smp: Brought up 1 node, 4 CPUs [ 0.059959] smpboot: Max logical packages: 1 [ 0.060012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.088771] node 0 deferred pages initialised in 26ms [ 0.092166] devtmpfs: initialized [ 0.093246] x86/mm: Memory block size: 128MB [ 0.096000] gcov: version magic: 0x41383552 [ 0.098171] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.099073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.100259] pinctrl core: initialized pinctrl subsystem [ 0.101187] [ 0.101673] ************************************************************* [ 0.102012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.103013] ** ** [ 0.104011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.105011] ** ** [ 0.106013] ** This means that this kernel is built to expose internal ** [ 0.107013] ** IOMMU data structures, which may compromise security on ** [ 0.108012] ** your system. ** [ 0.109011] ** ** [ 0.110016] ** If you see this message and you are not debugging the ** [ 0.111038] ** kernel, report this immediately to your vendor! ** [ 0.112014] ** ** [ 0.113015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114015] ************************************************************* [ 0.115623] NET: Registered protocol family 16 [ 0.116475] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.117057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.118056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.119381] cpuidle: using governor menu [ 0.121856] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.124555] PCI: Using configuration type 1 for base access [ 0.127142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.137055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.138038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.140167] cryptd: max_cpu_qlen set to 1000 [ 0.142267] ACPI: Added _OSI(Module Device) [ 0.143022] ACPI: Added _OSI(Processor Device) [ 0.144016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.145014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.149169] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.151569] ACPI: Interpreter enabled [ 0.152066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.153018] ACPI: Using IOAPIC for interrupt routing [ 0.154140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.155392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.165803] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.166043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.167021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.168084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.170368] acpiphp: Slot [2] registered [ 0.171132] acpiphp: Slot [5] registered [ 0.172142] acpiphp: Slot [6] registered [ 0.173097] acpiphp: Slot [7] registered [ 0.174128] acpiphp: Slot [8] registered [ 0.175125] acpiphp: Slot [9] registered [ 0.176140] acpiphp: Slot [10] registered [ 0.177126] acpiphp: Slot [3] registered [ 0.178115] acpiphp: Slot [4] registered [ 0.179074] acpiphp: Slot [11] registered [ 0.180137] acpiphp: Slot [12] registered [ 0.181111] acpiphp: Slot [13] registered [ 0.182149] acpiphp: Slot [14] registered [ 0.184200] acpiphp: Slot [15] registered [ 0.186139] acpiphp: Slot [16] registered [ 0.188150] acpiphp: Slot [17] registered [ 0.189141] acpiphp: Slot [18] registered [ 0.192196] acpiphp: Slot [19] registered [ 0.194172] acpiphp: Slot [20] registered [ 0.195148] acpiphp: Slot [21] registered [ 0.197218] acpiphp: Slot [22] registered [ 0.199141] acpiphp: Slot [23] registered [ 0.200156] acpiphp: Slot [24] registered [ 0.202140] acpiphp: Slot [25] registered [ 0.204246] acpiphp: Slot [26] registered [ 0.206150] acpiphp: Slot [27] registered [ 0.207149] acpiphp: Slot [28] registered [ 0.209149] acpiphp: Slot [29] registered [ 0.211153] acpiphp: Slot [30] registered [ 0.212150] acpiphp: Slot [31] registered [ 0.214166] PCI host bridge to bus 0000:00 [ 0.216118] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.219035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.222037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.225030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.228103] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.231050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.234254] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.238057] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.241331] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.252738] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.258014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.260021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.265024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.269663] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.271798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.274043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.277155] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.283019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.295017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.300879] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.305892] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.319014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.327014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.345016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.355000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.361019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.371067] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.391022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.401933] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.407021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.413016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.428015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.438627] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.446023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.454023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.474022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.492466] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.500023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.512020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.534022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.547095] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.552015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.557014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.575015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.585900] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.588470] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.592591] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.594392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.597248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.604005] iommu: Default domain type: Passthrough [ 0.605499] SCSI subsystem initialized [ 0.607165] ACPI: bus type USB registered [ 0.609117] usbcore: registered new interface driver usbfs [ 0.611089] usbcore: registered new interface driver hub [ 0.613086] usbcore: registered new device driver usb [ 0.615186] pps_core: LinuxPPS API ver. 1 registered [ 0.617013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.620074] PTP clock support registered [ 0.623124] EDAC MC: Ver: 3.0.0 [ 0.625191] PCI: Using ACPI for IRQ routing [ 0.626968] NetLabel: Initializing [ 0.627013] NetLabel: domain hash size = 128 [ 0.628012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.629225] NetLabel: unlabeled traffic allowed by default [ 0.631160] vgaarb: loaded [ 0.633316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.634014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.642163] clocksource: Switched to clocksource kvm-clock [ 0.754145] VFS: Disk quotas dquot_6.6.0 [ 0.756084] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759411] *** VALIDATE ramfs *** [ 0.760843] *** VALIDATE hugetlbfs *** [ 0.762710] pnp: PnP ACPI init [ 0.765688] pnp: PnP ACPI: found 6 devices [ 0.792401] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.796617] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.800099] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.803433] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.807180] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.810212] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.814337] NET: Registered protocol family 2 [ 0.817660] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.823209] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.826727] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.832186] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.835506] TCP: Hash tables configured (established 65536 bind 65536) [ 0.838257] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.841283] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.844452] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.846989] NET: Registered protocol family 1 [ 0.850022] RPC: Registered named UNIX socket transport module. [ 0.852336] RPC: Registered udp transport module. [ 0.854203] RPC: Registered tcp transport module. [ 0.856161] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.859037] NET: Registered protocol family 44 [ 0.860733] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.862958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.864882] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.867136] PCI: CLS 0 bytes, default 64 [ 0.868884] Unpacking initramfs... [ 2.279542] debug: unmapping init [mem 0xffff89d33cc54000-0xffff89d33ffbffff] [ 2.285062] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.287362] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.290334] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.810231] Initialise system trusted keyrings [ 2.812273] Key type blacklist registered [ 2.814162] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.828126] zbud: loaded [ 2.832195] *** VALIDATE nfs *** [ 2.834035] *** VALIDATE nfs4 *** [ 2.836222] pstore: using deflate compression [ 2.839807] Platform Keyring initialized [ 2.962458] NET: Registered protocol family 38 [ 2.963924] Key type asymmetric registered [ 2.965683] Asymmetric key parser 'x509' registered [ 2.967370] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.970306] io scheduler mq-deadline registered [ 2.971838] io scheduler kyber registered [ 2.973216] io scheduler bfq registered [ 2.974910] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.977621] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.980580] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.982978] ACPI: Power Button [PWRF] [ 2.988918] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.996752] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.010839] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.021872] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.041419] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.070253] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.100490] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.105328] Non-volatile memory driver v1.3 [ 3.107858] Linux agpgart interface v0.103 [ 3.142626] virtio_blk virtio1: [vda] 145416 512-byte logical blocks (74.5 MB/71.0 MiB) [ 3.145248] vda: detected capacity change from 0 to 74452992 [ 3.171713] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.174755] vdb: detected capacity change from 0 to 1073741824 [ 3.191053] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.193666] vdc: detected capacity change from 0 to 2621440000 [ 3.209775] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.213035] vdd: detected capacity change from 0 to 2621440000 [ 3.229737] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.233090] vde: detected capacity change from 0 to 4294967296 [ 3.251153] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.253793] vdf: detected capacity change from 0 to 4294967296 [ 3.261764] libphy: Fixed MDIO Bus: probed [ 3.266687] usbcore: registered new interface driver usbserial_generic [ 3.269402] usbserial: USB Serial support registered for generic [ 3.271752] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.276143] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.278177] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.280620] mousedev: PS/2 mouse device common for all mice [ 3.283684] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.284529] rtc_cmos 00:05: RTC can wake from S4 [ 3.290618] rtc_cmos 00:05: registered as rtc0 [ 3.290635] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.292537] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.298042] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.298714] intel_pstate: CPU model not supported [ 3.304667] hid: raw HID events driver (C) Jiri Kosina [ 3.307119] usbcore: registered new interface driver usbhid [ 3.309139] usbhid: USB HID core driver [ 3.310754] drop_monitor: Initializing network drop monitor service [ 3.313362] Initializing XFRM netlink socket [ 3.315643] NET: Registered protocol family 10 [ 3.318816] Segment Routing with IPv6 [ 3.320370] NET: Registered protocol family 17 [ 3.322736] mpls_gso: MPLS GSO support [ 3.328214] RAS: Correctable Errors collector initialized. [ 3.330517] AVX version of gcm_enc/dec engaged. [ 3.332027] AES CTR mode by8 optimization enabled [ 3.407977] sched_clock: Marking stable (3407946461, 0)->(4371766626, -963820165) [ 3.411771] registered taskstats version 1 [ 3.414489] Loading compiled-in X.509 certificates [ 3.416833] zswap: loaded using pool lzo/zbud [ 3.438865] Key type big_key registered [ 3.451934] Key type encrypted registered [ 3.453349] ima: No TPM chip found, activating TPM-bypass! [ 3.455185] ima: Allocated hash algorithm: sha1 [ 3.456849] ima: No architecture policies found [ 3.458659] evm: Initialising EVM extended attributes: [ 3.460828] evm: security.selinux [ 3.462256] evm: security.ima [ 3.463152] evm: security.capability [ 3.464513] evm: HMAC attrs: 0x1 [ 3.466741] rtc_cmos 00:05: setting system clock to 2026-08-15 18:54:23 UTC (1786820063) [ 3.473193] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.476171] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.484162] Write protecting the kernel read-only data: 28672k [ 3.487420] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.490030] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.525534] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.534175] systemd[1]: Detected virtualization kvm. [ 3.536149] systemd[1]: Detected architecture x86-64. [ 3.538107] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.567320] systemd[1]: No hostname configured. [ 3.569576] systemd[1]: Set hostname to . [ 3.573420] random: systemd: uninitialized urandom read (16 bytes read) [ 3.576320] systemd[1]: Initializing machine ID from random generator. [ 3.627105] random: ln: uninitialized urandom read (6 bytes read) [ 3.710427] random: systemd: uninitialized urandom read (16 bytes read) [ 3.713541] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.718119] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.722352] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.285871] device-mapper: uevent: version 1.0.3 [ 4.289163] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.977130] random: fast init done [ 5.027346] virtio_net virtio0 ens2: renamed from eth0 [ 5.074681] scsi host0: ata_piix [ 5.094103] scsi host1: ata_piix [ 5.095925] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.098289] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.676526] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.970945] random: crng init done [ 9.972234] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.285244] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.443443] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.715552] SELinux: Disabled at runtime. [ 11.786743] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.794011] systemd[1]: Detected virtualization kvm. [ 11.795656] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.280492] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.284227] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.289779] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.294244] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.297760] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.306239] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.313157] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Startin[ 12.338617] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.691132] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.028469] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.035509] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.134536] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.156257] EDAC sbridge: Ver: 1.1.2 [ 14.837106] Key type dns_resolver registered [ 15.137877] NFS: Registering the id_resolver key type [ 15.140487] Key type id_resolver registered [ 15.143549] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg414-server login: [ 39.849785] hrtimer: interrupt took 2906420 ns [ 59.378974] libcfs: loading out-of-tree module taints kernel. [ 59.425832] Key type ._llcrypt registered [ 59.428628] Key type .llcrypt registered [ 59.551496] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 77.718757] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 79.470869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.509382] alg: No test for adler32 (adler32-zlib) [ 80.982822] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 81.678198] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 83.471192] Key type lgssc registered [ 85.901983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.121098] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.139552] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 158.260748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.311520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.670179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.712692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.782542] Lustre: lustre-MDT0000: new disk, initializing [ 159.870297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.894921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.989720] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.320775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.442985] Lustre: 6521:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.525536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.537559] Lustre: Skipped 1 previous similar message [ 176.723854] Lustre: lustre-MDT0001: new disk, initializing [ 176.824808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.883618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.905111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.842787] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.193636] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.919494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.132603] Lustre: lustre-OST0000: new disk, initializing [ 199.139597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.147795] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.243278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.939172] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.903678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.925630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.041771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.467960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.623602] Lustre: lustre-OST0001: new disk, initializing [ 219.626491] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.631862] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.713436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.235993] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.974390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.985853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.023708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.118865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.789504] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 254.990501] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.581588] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 266.529908] Lustre: DEBUG MARKER: Client: 2.17.54.84 [ 270.077485] Lustre: DEBUG MARKER: MDS: 2.17.54.84 [ 272.743291] Lustre: DEBUG MARKER: OSS: 2.17.54.84 [ 274.197072] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 14:58:52 EDT 2026 [ 290.739153] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 292.121714] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 310.239716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.244930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.264058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.270634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.271150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.652419] Lustre: server umount lustre-MDT0000 complete [ 316.387724] LustreError: 7750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.406069] LustreError: 7750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 317.063716] LustreError: 7469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820377 with bad export cookie 10615800174178028779 [ 317.065481] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.070693] LustreError: 7469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 321.504636] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.504887] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.511613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.511624] Lustre: Skipped 3 previous similar messages [ 321.522823] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 321.558967] Lustre: Skipped 3 previous similar messages [ 322.528706] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.553407] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 323.408968] Lustre: server umount lustre-MDT0001 complete [ 327.110471] Lustre: server umount lustre-OST0000 complete [ 330.501691] Lustre: server umount lustre-OST0001 complete [ 336.785187] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 345.308304] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 381.057593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.349709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 381.386215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.496517] Lustre: lustre-MDT0000: new disk, initializing [ 381.560779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.591216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.710403] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 395.856792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.012227] Lustre: 15083:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.126751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.135262] Lustre: Skipped 1 previous similar message [ 396.252805] Lustre: lustre-MDT0001: new disk, initializing [ 396.374403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.431956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.441313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.787935] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 410.288228] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 415.995493] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 423.253772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.558270] Lustre: lustre-OST0000: new disk, initializing [ 423.562207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.568274] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.658324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.898430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 424.923452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.134543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.111533] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 440.085535] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 444.679873] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.883747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.347806] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.564252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.650388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.654310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.655009] LustreError: Skipped 1 previous similar message [ 455.660683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.665026] Lustre: Skipped 1 previous similar message [ 455.679393] Lustre: Skipped 4 previous similar messages [ 457.548488] Lustre: server umount lustre-OST0000 complete [ 462.816280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 462.825618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.838610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.307378] Lustre: server umount lustre-MDT0000 complete [ 470.582499] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820530 with bad export cookie 10615800174178030312 [ 470.583388] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.587461] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 470.934390] Lustre: server umount lustre-MDT0001 complete [ 474.845669] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:02:13 (1786820533) [ 477.223339] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 478.949255] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:02:17 (1786820537) [ 484.857137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.241665] LustreError: 19475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.316643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.248447] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 490.474718] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.199134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.736331] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 507.307527] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 513.527221] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 520.955290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.195987] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 521.201662] Lustre: Skipped 2 previous similar messages [ 527.158436] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 534.547669] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 545.766863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.767619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 545.769554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 545.779023] Lustre: Skipped 2 previous similar messages [ 548.077640] Lustre: server umount lustre-OST0000 complete [ 552.931117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.730183] Lustre: server umount lustre-MDT0000 complete [ 561.092322] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820621 with bad export cookie 10615800174178031194 [ 561.093875] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.102188] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 561.356861] Lustre: server umount lustre-MDT0001 complete [ 570.869456] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 573.552321] Key type lgssc unregistered [ 573.765402] LNet: 23007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.774265] LNetError: 23007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.788224] LNet: Removed LNI 192.168.204.114@tcp [ 574.629166] Key type .llcrypt unregistered [ 574.631063] Key type ._llcrypt unregistered [ 596.804876] Key type ._llcrypt registered [ 596.808267] Key type .llcrypt registered [ 596.939740] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 598.189600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 598.250799] alg: No test for adler32 (adler32-zlib) [ 599.315881] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 599.525669] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 601.215179] Key type lgssc registered [ 602.146624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 610.583811] loop: module loaded [ 610.673383] loop0: detected capacity change from 0 to 204800000 [ 610.905634] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.924900] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 611.416211] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.513456] loop0: detected capacity change from 0 to 204800000 [ 618.670422] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.681252] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.692028] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.922036] loop0: detected capacity change from 0 to 204800000 [ 630.120602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 630.158303] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.484464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 631.601627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 631.726488] Lustre: lustre-MDT0000: new disk, initializing [ 631.846442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.877323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 636.098425] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 644.504866] loop1: detected capacity change from 0 to 204800000 [ 650.008747] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 650.336637] Lustre: lustre-OST07c6: new disk, initializing [ 650.341708] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 650.352660] Lustre: Skipped 1 previous similar message [ 650.361623] Lustre: 26192:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 650.461638] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 656.810255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 658.082714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 658.090174] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 658.132556] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 662.527145] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 662.539487] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 662.547472] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 662.563993] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 662.576970] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 662.583656] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 678.263266] Lustre: 26831:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 678.273773] Lustre: 26831:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.281602] Lustre: 26831:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.289956] Lustre: 26831:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 678.299774] Lustre: 26831:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 678.304894] Lustre: 26831:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.142270] Lustre: 26831:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 683.151890] Lustre: 26831:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 683.160291] Lustre: 26831:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 683.173731] Lustre: 26831:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.185733] Lustre: 26831:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.191829] Lustre: 26831:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.202116] Lustre: 26831:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 683.216502] Lustre: 26831:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.227147] Lustre: 26831:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 683.236289] Lustre: 26831:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.242326] Lustre: 26831:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.251723] Lustre: 26831:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.888644] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 687.899552] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 687.905550] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 687.914669] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.921692] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 687.927595] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.934345] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 687.946610] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.959029] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 687.976670] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.983350] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 687.990540] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.786337] Lustre: 26831:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 695.802170] Lustre: 26831:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 695.813949] Lustre: 26831:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 695.820325] Lustre: 26831:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.834266] Lustre: 26831:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.839504] Lustre: 26831:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.848986] Lustre: 26831:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 695.871827] Lustre: 26831:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.882925] Lustre: 26831:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 695.892209] Lustre: 26831:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.899265] Lustre: 26831:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 695.903506] Lustre: 26831:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 705.673493] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 705.683507] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 705.693550] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 705.706646] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.716687] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 705.723881] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.733966] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 705.740640] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.746042] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 705.749919] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.753338] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.757020] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.066565] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.094684] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 755.180903] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 755.564318] Lustre: server umount lustre-OST07c6 complete [ 764.913655] Lustre: server umount lustre-MDT0000 complete [ 780.016051] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 782.378483] Key type lgssc unregistered [ 782.594188] LNet: 28748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.610769] LNetError: 28748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.629936] LNet: Removed LNI 192.168.204.114@tcp [ 783.265562] Key type .llcrypt unregistered [ 783.267858] Key type ._llcrypt unregistered [ 800.765704] Key type ._llcrypt registered [ 800.769376] Key type .llcrypt registered [ 800.890610] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 802.341729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.361187] alg: No test for adler32 (adler32-zlib) [ 803.387778] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 803.662822] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 805.367170] Key type lgssc registered [ 806.449711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 816.472829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 816.496956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.034916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.255880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 828.406366] LustreError: 30019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.428214] LustreError: 30019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 832.592124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.089326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 837.613508] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 846.048023] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 852.183729] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 860.068910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 860.419445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.554331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 866.891783] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 875.263467] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.442903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 894.444177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.455674] LustreError: Skipped 2 previous similar messages [ 894.475297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 894.481231] Lustre: Skipped 1 previous similar message [ 898.989490] Lustre: server umount lustre-OST0000 complete [ 904.673341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.674948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 904.675390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.684253] Lustre: Skipped 2 previous similar messages [ 909.795433] LustreError: 30747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.817941] LustreError: 30747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 909.941261] Lustre: server umount lustre-MDT0000 complete [ 914.274587] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820974 with bad export cookie 18418948430688591437 [ 914.282739] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.612372] Lustre: server umount lustre-MDT0001 complete [ 926.732680] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 929.578580] Key type lgssc unregistered [ 930.003831] LNet: 33760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.018581] LNetError: 33760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.038198] LNet: Removed LNI 192.168.204.114@tcp [ 931.218958] Key type .llcrypt unregistered [ 931.222227] Key type ._llcrypt unregistered [ 949.653558] Key type ._llcrypt registered [ 949.656295] Key type .llcrypt registered [ 949.735692] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 950.904662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.915827] alg: No test for adler32 (adler32-zlib) [ 952.213956] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 952.462149] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 954.259585] Key type lgssc registered [ 955.657380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 963.991948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 964.025531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.661419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.995476] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 975.847773] LustreError: 35030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.866101] LustreError: 35030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 980.625049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.961320] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 981.238993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 985.633278] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 994.784523] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1000.799434] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1008.194927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.585884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.611329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1014.810785] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1022.648575] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1036.261158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.269567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.276764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.283169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.283187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.293145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.118292] Lustre: server umount lustre-OST0000 complete [ 1047.526854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.530676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.545914] Lustre: Skipped 1 previous similar message [ 1047.558511] Lustre: Skipped 2 previous similar messages [ 1051.880244] Lustre: server umount lustre-MDT0000 complete [ 1052.640660] LustreError: 35031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.655879] LustreError: 35031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1055.613201] LustreError: 35011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821115 with bad export cookie 16114497934713791890 [ 1055.618886] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1055.623717] LustreError: 35011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1056.028953] Lustre: server umount lustre-MDT0001 complete [ 1066.959832] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1069.589659] Key type lgssc unregistered [ 1069.892090] LNet: 38585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.901827] LNetError: 38585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.918530] LNet: Removed LNI 192.168.204.114@tcp [ 1070.825181] Key type .llcrypt unregistered [ 1070.829125] Key type ._llcrypt unregistered [ 1093.134464] Key type ._llcrypt registered [ 1093.138658] Key type .llcrypt registered [ 1093.280415] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1094.510872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.568876] alg: No test for adler32 (adler32-zlib) [ 1095.627470] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1095.855383] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1097.511212] Key type lgssc registered [ 1098.616419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.390811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.402808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.808750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.768781] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1117.156394] LustreError: 40004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.167330] LustreError: 40004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1118.063830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.367299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.923724] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1128.599595] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1134.308815] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1140.993688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.278869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.687811] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1152.555152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1154.701771] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1161.660722] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 1162.528301] Lustre: server umount lustre-MDT0000 complete [ 1162.733304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.739760] LustreError: 41102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.748920] Lustre: Skipped 1 previous similar message [ 1162.774899] LustreError: 41102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1166.075162] LustreError: 40740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821226 with bad export cookie 13938511275164715100 [ 1166.079272] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.084583] LustreError: 40740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1166.742576] LustreError: 39999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.761606] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 1167.840727] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.848072] Lustre: Skipped 1 previous similar message [ 1167.851363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1171.866048] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 1171.866048] LustreError: 39999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.866057] LustreError: 39999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1172.535295] Lustre: server umount lustre-MDT0001 complete [ 1193.348081] Lustre: server umount lustre-OST0000 complete [ 1206.710597] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1209.623799] Key type lgssc unregistered [ 1209.858584] LNet: 43863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.866301] LNetError: 43863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.883451] LNet: Removed LNI 192.168.204.114@tcp [ 1210.701098] Key type .llcrypt unregistered [ 1210.702980] Key type ._llcrypt unregistered [ 1234.262219] Key type ._llcrypt registered [ 1234.264788] Key type .llcrypt registered [ 1234.368222] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1235.248192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1235.288118] alg: No test for adler32 (adler32-zlib) [ 1236.589621] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1236.991477] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1238.671212] Key type lgssc registered [ 1240.035924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1246.734635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1246.746276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.181296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.448412] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1258.486127] LustreError: 45279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.512628] LustreError: 45279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1261.069925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.473059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.458080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1272.861646] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.812844] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1285.735298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1286.147134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.561953] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1293.308378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1300.790455] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1308.643996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.644408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.693864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.709915] Lustre: Skipped 1 previous similar message [ 1313.607985] Lustre: server umount lustre-OST0000 complete [ 1333.218109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.222782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1333.227837] Lustre: Skipped 1 previous similar message [ 1333.243828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.256192] Lustre: Skipped 2 previous similar messages [ 1337.395509] Lustre: server umount lustre-MDT0000 complete [ 1338.354452] LustreError: 45274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.374319] LustreError: 45274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1341.556026] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821401 with bad export cookie 13187347574818990616 [ 1341.563617] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.572688] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1347.999682] Lustre: server umount lustre-MDT0001 complete [ 1359.478921] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1362.263707] Key type lgssc unregistered [ 1362.585627] LNet: 48986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.594530] LNetError: 48986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.617348] LNet: Removed LNI 192.168.204.114@tcp [ 1363.523270] Key type .llcrypt unregistered [ 1363.524912] Key type ._llcrypt unregistered [ 1386.602781] Key type ._llcrypt registered [ 1386.604700] Key type .llcrypt registered [ 1386.710412] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1387.551932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1387.570938] alg: No test for adler32 (adler32-zlib) [ 1388.676465] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1388.944276] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1390.639168] Key type lgssc registered [ 1391.494539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1399.167309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1399.181126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.595742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.452136] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1410.023991] LustreError: 50402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.037250] LustreError: 50402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1413.789033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.127445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.799593] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1426.603594] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1432.135369] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1438.648713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.980572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.927529] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1445.123687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1452.789388] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1457.867780] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1461.552558] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1473.666866] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1515h18m51s [ 1475.323862] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1480.116123] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 1480.673099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1480.680170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.692619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.763263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.774655] Lustre: Skipped 5 previous similar messages [ 1484.161915] Lustre: server umount lustre-MDT0000 complete [ 1486.229186] LustreError: 52245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.775227] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1487.845606] LustreError: 50403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.858514] LustreError: 50403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1492.382892] LustreError: 52245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.192594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.274173] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.522848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.575567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1495.536825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1498.738065] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1501.949363] Lustre: DEBUG MARKER: Wait for df (36514) ... [ 1503.290827] Lustre: DEBUG MARKER: done [ 1506.785335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.793927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.807284] Lustre: Skipped 2 previous similar messages [ 1506.812829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.858329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.865667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.881623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1510.898319] Lustre: Skipped 2 previous similar messages [ 1512.410211] Lustre: server umount lustre-OST0000 complete [ 1521.120519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.120942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1521.123160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.127722] Lustre: Skipped 1 previous similar message [ 1531.359181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1531.514296] Lustre: server umount lustre-MDT0000 complete [ 1534.434675] LustreError: 50402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.472236] LustreError: 50402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1535.245955] LustreError: 52248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821595 with bad export cookie 8324071132184393581 [ 1535.250366] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.253138] LustreError: 52248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1535.537972] Lustre: server umount lustre-MDT0001 complete [ 1545.907236] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1548.415729] Key type lgssc unregistered [ 1548.679387] LNet: 55331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.686929] LNetError: 55331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.712990] LNet: Removed LNI 192.168.204.114@tcp [ 1549.505673] Key type .llcrypt unregistered [ 1549.508858] Key type ._llcrypt unregistered [ 1590.723561] Key type ._llcrypt registered [ 1590.730905] Key type .llcrypt registered [ 1590.892839] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1592.219810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.264070] alg: No test for adler32 (adler32-zlib) [ 1593.304379] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1593.574580] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1595.263217] Key type lgssc registered [ 1596.397493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.853445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.875841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.100469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1605.124456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1605.289262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.968218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1617.021912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.093204] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1617.116146] Lustre: 57540:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1617.414211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1621.636856] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1629.070904] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.024425] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1640.466895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1640.719855] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1641.004741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.198822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1647.643410] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1654.418578] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1660.791242] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1515h21m58s [ 1662.538905] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1670.281406] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1672.953909] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1673.001601] LustreError: 57548:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1693.156206] Lustre: lustre-MDT0000: Client c31d7351-8f46-4990-af46-e8567dd23e14 (at 192.168.204.14@tcp) reconnecting [ 1713.023160] LustreError: 57548:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1715.819879] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1719.191081] Lustre: DEBUG MARKER: done [ 1728.996615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.999036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.026322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.045408] Lustre: Skipped 1 previous similar message [ 1732.732325] Lustre: server umount lustre-OST0000 complete [ 1739.233040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.237256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.249101] Lustre: Skipped 2 previous similar messages [ 1739.260454] Lustre: Skipped 1 previous similar message [ 1740.321279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.330578] Lustre: Skipped 1 previous similar message [ 1743.408351] Lustre: server umount lustre-MDT0000 complete [ 1747.758051] LustreError: 57534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821807 with bad export cookie 7319507493952197024 [ 1747.769766] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.779457] LustreError: 57534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1748.153652] Lustre: server umount lustre-MDT0001 complete [ 1760.384375] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1762.984812] Key type lgssc unregistered [ 1763.192975] LNet: 61871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.204345] LNetError: 61871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.221658] LNet: Removed LNI 192.168.204.114@tcp [ 1763.956154] Key type .llcrypt unregistered [ 1763.957717] Key type ._llcrypt unregistered [ 1807.259422] Key type ._llcrypt registered [ 1807.263905] Key type .llcrypt registered [ 1807.384898] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1808.746740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.882489] alg: No test for adler32 (adler32-zlib) [ 1810.023935] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 1810.286569] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1811.999179] Key type lgssc registered [ 1813.297644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1817.184195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1817.200562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.462501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1818.493340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1818.726207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.817784] Lustre: Failing over lustre-MDT0000 [ 1820.029272] Lustre: server umount lustre-MDT0000 complete [ 1826.963894] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:24:45 (1786821885) [ 1833.459548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.917673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.841353] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1847.535524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.659233] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1847.701742] Lustre: 64673:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1847.995390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1851.862314] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1859.890884] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.717573] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1872.237387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1872.374177] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1872.565523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.162986] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1880.322584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1886.879299] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.870527] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1898.981671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1898.990287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.003722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.025910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.039210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.811913] Lustre: server umount lustre-MDT0000 complete [ 1909.401190] LustreError: 64664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821969 with bad export cookie 18025658885247192140 [ 1909.415054] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.422736] LustreError: 64664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1909.740577] Lustre: server umount lustre-MDT0001 complete [ 1914.236461] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1919.528982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1927.650980] Lustre: 63594:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821971/real 1786821971] req@0000000061351167 x1873618529198208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786821987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1927.693310] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.721689] Lustre: Skipped 1 previous similar message [ 1928.574613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.719418] Lustre: 63597:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821974/real 1786821974] req@00000000472f0e2c x1873618529198336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786821990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1932.768179] Lustre: 63596:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821976/real 1786821976] req@00000000a18cfaaf x1873618529198464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786821992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1934.946120] LustreError: 63593:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d0f3957f x1873618529199232/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1935.325762] LustreError: 66519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.339704] LustreError: 66519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1935.397730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.481467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1935.903223] Lustre: 63594:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821979/real 1786821979] req@0000000083b7fdfa x1873618529198720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786821995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1936.416816] LustreError: 66520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.450600] LustreError: 68438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.467791] LustreError: 68438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1938.828565] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1941.471859] Lustre: 63594:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821984/real 1786821984] req@000000005c41aeac x1873618529198848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786822000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1942.500726] LustreError: 68601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.526321] LustreError: 68601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1945.723243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.059440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1949.725608] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1951.209992] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1955.792519] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1956.329545] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1961.473503] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1971.684340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.686753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.693189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.693197] Lustre: Skipped 2 previous similar messages [ 1971.727204] Lustre: Skipped 2 previous similar messages [ 1974.734785] Lustre: server umount lustre-MDT0000 complete [ 1976.805343] LustreError: 68440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.825684] LustreError: 68440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1978.747813] LustreError: 66525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822038 with bad export cookie 18025658885247194415 [ 1978.759649] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.777857] LustreError: 66525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1979.119499] Lustre: server umount lustre-MDT0001 complete [ 1986.419129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1996.608625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.279344] Lustre: 63594:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786822041/real 1786822041] req@00000000d530ced2 x1873618529224832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786822057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1997.306555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.487811] LustreError: 63593:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000535c0447 x1873618529225856/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.792244] LustreError: 69732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.892132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.981316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2009.062286] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2018.575612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.334479] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2024.427096] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2026.489375] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2031.109915] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2036.520761] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2043.503370] Lustre: server umount lustre-MDT0000 complete [ 2044.395526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2044.408417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.432168] LustreError: 66519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.442119] LustreError: 66519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2047.306978] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822107 with bad export cookie 18025658885247195794 [ 2047.313187] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.314775] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.664573] Lustre: server umount lustre-MDT0001 complete [ 2055.154753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2058.871537] Lustre: DEBUG MARKER: files compared the same [ 2070.331177] Lustre: server umount lustre-OST0000 complete [ 2091.521376] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2094.730714] Key type lgssc unregistered [ 2095.103699] LNet: 74396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.112863] LNetError: 74396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.130039] LNet: Removed LNI 192.168.204.114@tcp [ 2096.144678] Key type .llcrypt unregistered [ 2096.147459] Key type ._llcrypt unregistered [ 2122.033648] Key type ._llcrypt registered [ 2122.037141] Key type .llcrypt registered [ 2122.117318] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2123.485597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.558745] alg: No test for adler32 (adler32-zlib) [ 2124.601336] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2124.903606] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2126.607390] Key type lgssc registered [ 2127.401740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.195380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2135.219488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.930738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.537981] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug +malloc all [ 2146.282189] LustreError: 75815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.301098] LustreError: 75815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2151.397065] LustreError: 75816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.907875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.276616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.567692] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug +malloc all [ 2167.156572] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2173.936512] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2181.802114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.221866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.045561] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug +malloc all [ 2197.204698] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2206.760335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2216.929685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.930738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.962100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.980698] Lustre: Skipped 1 previous similar message [ 2222.050248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.100682] Lustre: server umount lustre-OST0000 complete [ 2230.242203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.255888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.275758] Lustre: Skipped 1 previous similar message [ 2230.286558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.294930] Lustre: Skipped 1 previous similar message [ 2232.288678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.289041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.986656] Lustre: server umount lustre-MDT0000 complete [ 2237.408062] LustreError: 75815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.438486] LustreError: 75815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2238.017250] LustreError: 75795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822298 with bad export cookie 14082695914305141353 [ 2238.034908] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.039613] LustreError: 75795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2238.450892] Lustre: server umount lustre-MDT0001 complete [ 2251.347425] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2254.742477] Key type lgssc unregistered [ 2255.118723] LNet: 79370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.132086] LNetError: 79370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.161479] LNet: Removed LNI 192.168.204.114@tcp [ 2256.123567] Key type .llcrypt unregistered [ 2256.127290] Key type ._llcrypt unregistered [ 2280.127986] Key type ._llcrypt registered [ 2280.130711] Key type .llcrypt registered [ 2280.253515] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2281.368290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.506431] alg: No test for adler32 (adler32-zlib) [ 2282.639186] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2282.920084] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2284.599200] Key type lgssc registered [ 2285.908227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2294.116548] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2294.152733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.727428] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2314.848963] LustreError: 80761:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 2341.535205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.360759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2354.695795] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2361.251993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.392929] Lustre: *** cfs_fail_loc=706, val=0*** [ 2361.699846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.783259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2366.189527] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2366.945918] LustreError: 80786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.954885] LustreError: 80786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2372.066523] LustreError: 80786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.046885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.385410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.406601] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2384.690707] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2390.031482] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.050058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.058578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.064665] Lustre: Skipped 1 previous similar message [ 2392.081891] Lustre: Skipped 1 previous similar message [ 2397.158525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.165397] Lustre: Skipped 1 previous similar message [ 2402.275387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.283097] Lustre: Skipped 1 previous similar message [ 2406.367737] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2406.470516] Lustre: server umount lustre-OST0000 complete [ 2412.525607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.548474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.566048] Lustre: Skipped 1 previous similar message [ 2417.219958] Lustre: server umount lustre-MDT0000 complete [ 2417.648311] LustreError: 83476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.704411] LustreError: 83476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2422.752902] LustreError: 81847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.765965] LustreError: 81847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2422.780660] LustreError: 80769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822482 with bad export cookie 16278403777063106716 [ 2422.795540] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2422.797889] LustreError: 80769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2423.350102] Lustre: server umount lustre-MDT0001 complete [ 2434.455863] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2437.251447] Key type lgssc unregistered [ 2437.498820] LNet: 84378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.505551] LNetError: 84378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.525441] LNet: Removed LNI 192.168.204.114@tcp [ 2438.210973] Key type .llcrypt unregistered [ 2438.213525] Key type ._llcrypt unregistered [ 2461.131794] Key type ._llcrypt registered [ 2461.133773] Key type .llcrypt registered [ 2461.198378] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2461.681385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.759152] alg: No test for adler32 (adler32-zlib) [ 2462.846578] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2463.083518] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2464.751282] Key type lgssc registered [ 2466.006941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.578069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.587573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.855237] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2488.317514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.826660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.814951] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2493.922776] LustreError: 86461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.945317] LustreError: 86461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2498.025123] LustreError: 86465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.046381] LustreError: 86465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2501.145736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2501.384590] LustreError: 86464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.390644] LustreError: 86464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2501.412846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.722803] LustreError: 86459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.774331] LustreError: 86459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2508.608432] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2511.847926] LustreError: 86465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.868372] LustreError: 86465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2517.623188] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.634341] Lustre: MGS: Client 4952cb0c-da5a-4be5-b312-9e67c2b3cd43 (at 0@lo) reconnecting [ 2517.649692] LustreError: 86475:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2517.849946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.948666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2522.399401] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2538.471445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.474807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.499845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.515130] Lustre: Skipped 1 previous similar message [ 2542.087556] Lustre: server umount lustre-OST0000 complete [ 2548.705716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.709114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2548.720648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.733159] Lustre: Skipped 2 previous similar messages [ 2552.092074] Lustre: server umount lustre-MDT0000 complete [ 2553.828058] LustreError: 88004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.843327] LustreError: 88004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2555.811578] Lustre: server umount lustre-MDT0001 complete [ 2558.982925] Lustre: server umount MGS complete [ 2569.051540] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:37:07 (1786822627) [ 2596.100267] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2606.489941] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2646.851364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.300458] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2663.853227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.155582] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2664.223598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2664.304097] Lustre: lustre-MDT0001: new disk, initializing [ 2664.378301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2664.398317] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2664.401606] Lustre: 93630:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2664.412777] LustreError: 93630:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2669.748067] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2680.079151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2680.273398] Lustre: lustre-OST0000: new disk, initializing [ 2680.276896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2680.282543] Lustre: 94399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2680.339560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.835963] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2690.086513] Lustre: 94874:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d06f6d8a x1873619213918464/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786822766 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2694.972896] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2701.704701] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2701.725947] Lustre: MGS: Client efcc57f6-97aa-49b3-ad21-dfb755a7e8f8 (at 0@lo) reconnecting [ 2701.743101] LustreError: 93616:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2701.785917] Lustre: 92898:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2701.835868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2701.904317] Lustre: lustre-MDT0000: new disk, initializing [ 2702.090423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.124458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2702.289516] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2703.200041] Lustre: 95467:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000adc9fa59 x1873619213925248/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786822779 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2707.073302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2710.516651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2710.534319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2712.305452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2712.311667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2712.430645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2738.148235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.157046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.171227] Lustre: Skipped 1 previous similar message [ 2738.174915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.200524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.201483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.843434] Lustre: server umount lustre-OST0000 complete [ 2750.439032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.441854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.453575] Lustre: Skipped 1 previous similar message [ 2750.465200] Lustre: Skipped 2 previous similar messages [ 2752.481946] Lustre: server umount lustre-MDT0000 complete [ 2755.557421] LustreError: 93601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.583825] LustreError: 93601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2762.956956] Lustre: server umount lustre-MDT0001 complete [ 2767.240285] Lustre: server umount MGS complete [ 2778.433457] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:40:36 (1786822836) [ 2798.730575] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2802.703519] Key type lgssc unregistered [ 2803.164809] LNet: 97967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.186892] LNetError: 97967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2803.209220] LNet: Removed LNI 192.168.204.114@tcp [ 2804.308337] Key type .llcrypt unregistered [ 2804.311401] Key type ._llcrypt unregistered [ 2822.895445] Key type ._llcrypt registered [ 2822.897594] Key type .llcrypt registered [ 2823.005385] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2824.214346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2824.223450] alg: No test for adler32 (adler32-zlib) [ 2825.391630] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2825.638164] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2827.311169] Key type lgssc registered [ 2828.571331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2835.708069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2835.726614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.144652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.060390] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2847.716283] LustreError: 99194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.740375] LustreError: 99194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2849.372706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.715333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.946557] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2860.906410] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.722361] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2872.174509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2872.504222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.470431] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2882.742093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2885.334285] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2893.287260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.294866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.315347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.315588] Lustre: Skipped 1 previous similar message [ 2898.414542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.425291] Lustre: Skipped 1 previous similar message [ 2899.270219] Lustre: server umount lustre-OST0000 complete [ 2906.082099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2906.095538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.108035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.112818] Lustre: Skipped 1 previous similar message [ 2908.640750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.651647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.765381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.778720] Lustre: Skipped 2 previous similar messages [ 2918.367786] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2918.477889] Lustre: server umount lustre-MDT0000 complete [ 2918.881195] LustreError: 99194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.902502] LustreError: 99194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2922.093880] LustreError: 99175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822982 with bad export cookie 2038561417515872099 [ 2922.099394] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.102616] LustreError: 99175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2922.359435] Lustre: server umount lustre-MDT0001 complete [ 2933.788481] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2936.825955] Key type lgssc unregistered [ 2937.067724] LNet: 102725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.080152] LNetError: 102725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2937.105274] LNet: Removed LNI 192.168.204.114@tcp [ 2937.926117] Key type .llcrypt unregistered [ 2937.934854] Key type ._llcrypt unregistered [ 2952.646847] Key type ._llcrypt registered [ 2952.649088] Key type .llcrypt registered [ 2952.747342] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2953.711544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2953.727452] alg: No test for adler32 (adler32-zlib) [ 2954.823589] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 2955.066833] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2956.751951] Key type lgssc registered [ 2957.920766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2963.460741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2963.473994] LustreError: 103797:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2963.505602] LustreError: 103809:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2963.510330] LustreError: 103797:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4967 [ 2963.514778] LustreError: 103797:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2963.525249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.526268] LustreError: 103797:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2965.071358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.243180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.636716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2976.852519] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2981.693772] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing lsmod [ 2984.286435] LustreError: 105006:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2984.371779] LustreError: 105016:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2984.378838] LustreError: 105006:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4914 [ 2984.387854] LustreError: 105006:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2984.395137] LustreError: 105006:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2984.405364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.873547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.002335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.008828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.017110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.051995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.057860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.880390] Lustre: server umount lustre-MDT0000 complete [ 2995.169689] LustreError: 103842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.198715] LustreError: 103842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2997.087971] LustreError: 103824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823057 with bad export cookie 568802545326476427 [ 2997.092926] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.096126] LustreError: 103824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2997.115512] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.117095] LustreError: 105276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.126395] Lustre: Skipped 1 previous similar message [ 2997.138795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2997.168095] Lustre: Skipped 2 previous similar messages [ 2997.384099] Lustre: server umount lustre-MDT0001 complete [ 3001.071378] Lustre: server umount lustre-OST0000 complete [ 3009.069503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.464536] LustreError: 106079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.542988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.075727] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3014.639076] LustreError: 106080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.744900] LustreError: 106079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.773544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.114914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3026.849426] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3035.387945] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3043.393871] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3051.381772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.687406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.769852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3058.121750] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3066.612857] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3075.557585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3075.576770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.608159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.662898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3083.747392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.759223] Lustre: Skipped 2 previous similar messages [ 3089.375287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3089.511967] Lustre: server umount lustre-OST0000 complete [ 3093.988986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.000457] Lustre: Skipped 1 previous similar message [ 3094.003351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3094.007914] Lustre: Skipped 4 previous similar messages [ 3099.892836] Lustre: server umount lustre-MDT0000 complete [ 3104.224579] LustreError: 106075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.254416] LustreError: 106075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3104.679480] LustreError: 106061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823164 with bad export cookie 568802545326477134 [ 3104.687605] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.700508] LustreError: 106061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3105.036456] Lustre: server umount lustre-MDT0001 complete [ 3117.171758] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3120.121689] Key type lgssc unregistered [ 3120.380767] LNet: 109613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3120.399565] LNetError: 109613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3120.428595] LNet: Removed LNI 192.168.204.114@tcp [ 3121.505276] Key type .llcrypt unregistered [ 3121.525219] Key type ._llcrypt unregistered [ 3148.197096] Key type ._llcrypt registered [ 3148.198641] Key type .llcrypt registered [ 3148.276134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3149.516443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.554198] alg: No test for adler32 (adler32-zlib) [ 3150.558857] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 3150.770030] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3152.463331] Key type lgssc registered [ 3153.692295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3162.407763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3162.454980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.856950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.840175] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3173.354911] LustreError: 111033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.381705] LustreError: 111033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3178.076626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.549507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3183.568545] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3191.827529] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3198.820527] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3205.579790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.881631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.976912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3212.095661] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3219.714850] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.654555] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3231.357782] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3238.882154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.886248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3238.893740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.919816] Lustre: Skipped 1 previous similar message [ 3238.941848] Lustre: Skipped 1 previous similar message [ 3241.311809] Lustre: server umount lustre-OST0000 complete [ 3245.024140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3245.028690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.037386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.122776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.124424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.150040] Lustre: Skipped 1 previous similar message [ 3254.245090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.249891] Lustre: Skipped 1 previous similar message [ 3259.362116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.363177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 3259.638574] Lustre: server umount lustre-MDT0000 complete [ 3264.328853] LustreError: 111012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823324 with bad export cookie 12919675173269468172 [ 3264.329609] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.351341] LustreError: 111012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3264.913210] Lustre: server umount lustre-MDT0001 complete [ 3280.510991] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3283.509158] Key type lgssc unregistered [ 3283.738537] LNet: 114682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3283.744929] LNetError: 114682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3283.766098] LNet: Removed LNI 192.168.204.114@tcp [ 3284.897465] Key type .llcrypt unregistered [ 3284.902657] Key type ._llcrypt unregistered [ 3302.489534] Key type ._llcrypt registered [ 3302.493635] Key type .llcrypt registered [ 3302.613139] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3303.553057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.575936] alg: No test for adler32 (adler32-zlib) [ 3304.644837] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 3304.933355] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3306.735375] Key type lgssc registered [ 3308.106896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3316.925709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3316.945899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.715903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.462213] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3327.970671] LustreError: 115904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.015416] LustreError: 115904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3333.089570] LustreError: 115905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.159509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.460355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.734879] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3347.539404] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3353.164486] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3360.052726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.462498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.581171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3367.435177] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3374.524702] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3387.879065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3387.885466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.898084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3390.433606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3390.437843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3390.452287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3392.276214] Lustre: server umount lustre-OST0000 complete [ 3400.690817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.695056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.720189] Lustre: Skipped 1 previous similar message [ 3400.741551] Lustre: Skipped 2 previous similar messages [ 3404.047390] Lustre: server umount lustre-MDT0000 complete [ 3405.794949] LustreError: 117293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.817694] LustreError: 117293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3408.270263] LustreError: 115886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823468 with bad export cookie 12841287943860097261 [ 3408.279454] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3408.685368] Lustre: server umount lustre-MDT0001 complete [ 3422.438647] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3425.970315] Key type lgssc unregistered [ 3426.279160] LNet: 119440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3426.288623] LNetError: 119440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3426.315752] LNet: Removed LNI 192.168.204.114@tcp [ 3427.248468] Key type .llcrypt unregistered [ 3427.250347] Key type ._llcrypt unregistered [ 3450.561830] Key type ._llcrypt registered [ 3450.565297] Key type .llcrypt registered [ 3450.717236] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3452.047410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3452.083271] alg: No test for adler32 (adler32-zlib) [ 3453.089673] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 3453.281764] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3454.967660] Key type lgssc registered [ 3456.823488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3465.963943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3466.005475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.693855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.031413] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3478.002714] LustreError: 120858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.026833] LustreError: 120858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3482.849932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.111328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3483.240950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.682487] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3497.661470] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3504.678910] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3512.568199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.816581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.854189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3519.885507] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3528.216444] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.328836] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3539.111675] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3539.116426] Lustre: Skipped 2 previous similar messages [ 3544.058581] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3544.063710] Lustre: Skipped 2 previous similar messages [ 3545.202981] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3545.239308] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3552.580958] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.14@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3552.588228] Lustre: Skipped 2 previous similar messages [ 3562.564934] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.14@tcp 192.168.0.[2,10]@tcp [ 3568.379441] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.14/24@tcp in log lustre-MDT0000 [ 3568.390482] Lustre: Skipped 2 previous similar messages [ 3568.838360] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.14/24@tcp [ 3568.845217] Lustre: Skipped 1 previous similar message [ 3580.390207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.401067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.427342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3584.481334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3584.491024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3584.499469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.337540] Lustre: server umount lustre-OST0000 complete [ 3590.629833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3590.635046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.648209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.656551] Lustre: Skipped 1 previous similar message [ 3594.721779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.725750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.748445] Lustre: Skipped 1 previous similar message [ 3597.113630] Lustre: server umount lustre-MDT0000 complete [ 3599.841745] LustreError: 120858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.866296] LustreError: 120858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3601.205134] LustreError: 123488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823661 with bad export cookie 2401435564627989584 [ 3601.211033] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3601.226830] LustreError: 123488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3601.660633] Lustre: server umount lustre-MDT0001 complete [ 3612.874376] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3615.859613] Key type lgssc unregistered [ 3616.171698] LNet: 125697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3616.178958] LNetError: 125697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3616.191310] LNet: Removed LNI 192.168.204.114@tcp [ 3617.073421] Key type .llcrypt unregistered [ 3617.075213] Key type ._llcrypt unregistered [ 3651.849468] Key type ._llcrypt registered [ 3651.854049] Key type .llcrypt registered [ 3651.990329] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3653.247722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3653.305417] alg: No test for adler32 (adler32-zlib) [ 3654.384674] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 3654.649839] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3656.375869] Key type lgssc registered [ 3657.779616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3666.248900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3666.270757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.874793] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3667.920328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.931868] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.14/24@tcp [ 3673.894629] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3677.155878] LustreError: 127259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.183597] LustreError: 127259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3682.280573] LustreError: 127260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.265532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.911753] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3685.986766] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.14/24@tcp [ 3686.093223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3693.157522] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3703.076859] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3710.186685] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3718.855664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.450343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.433694] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3734.327330] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3737.926565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3751.393062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3751.406695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.421623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.447764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.450451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.466046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3753.484403] Lustre: Skipped 1 previous similar message [ 3756.464970] Lustre: server umount lustre-OST0000 complete [ 3762.657067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3762.666447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.693484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3766.538987] Lustre: server umount lustre-MDT0000 complete [ 3768.801750] LustreError: 127259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.831054] LustreError: 127259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3770.407754] LustreError: 127240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823830 with bad export cookie 10663733543280482270 [ 3770.436222] LustreError: 127240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3770.436648] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.828473] Lustre: server umount lustre-MDT0001 complete [ 3781.322171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3784.352448] Key type lgssc unregistered [ 3784.697054] LNet: 130852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3784.704063] LNetError: 130852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3784.720980] LNet: Removed LNI 192.168.204.114@tcp [ 3785.611464] Key type .llcrypt unregistered [ 3785.613801] Key type ._llcrypt unregistered [ 3815.038962] Key type ._llcrypt registered [ 3815.040776] Key type .llcrypt registered [ 3815.132108] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3829.302825] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3831.054042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3831.090208] alg: No test for adler32 (adler32-zlib) [ 3832.226875] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 3832.549966] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3834.295170] Key type lgssc registered [ 3835.617895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3873.492592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3873.527171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.827339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3874.849654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3874.898317] Lustre: lustre-MDT0000: new disk, initializing [ 3874.963122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.974966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3879.582639] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3889.998809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.097605] Lustre: 134253:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3890.133108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3890.136861] Lustre: Skipped 1 previous similar message [ 3890.183338] Lustre: lustre-MDT0001: new disk, initializing [ 3890.238207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3890.264601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3890.274582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3896.080566] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3906.145732] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3912.860036] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3921.422798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3921.894172] Lustre: lustre-OST0000: new disk, initializing [ 3921.911049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3921.923501] Lustre: 136221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3922.072438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.649650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3927.671926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3927.832045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3928.702715] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3940.632761] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3947.739951] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3948.052265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3953.452070] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3953.616868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3958.756665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3958.758886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.769183] LustreError: Skipped 1 previous similar message [ 3958.770252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3958.784883] Lustre: Skipped 1 previous similar message [ 3958.806751] Lustre: Skipped 1 previous similar message [ 3962.079643] Lustre: server umount lustre-OST0000 complete [ 3966.945321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3966.953066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.971787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.991823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.999380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.762021] Lustre: server umount lustre-MDT0000 complete [ 3977.248144] LustreError: 135057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824037 with bad export cookie 2918466673556612361 [ 3977.254396] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3977.259854] LustreError: 135057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3977.621880] Lustre: server umount lustre-MDT0001 complete [ 3988.850710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.921069] LustreError: 138441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3989.952049] LustreError: 138441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3990.091605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.105905] LustreError: 138442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.143158] LustreError: 138442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3995.320538] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3999.200638] LustreError: 138441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.321189] LustreError: 138442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.331592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.748634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4011.114700] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4020.543878] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4027.575874] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4034.760523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4035.143973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.766876] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4049.824537] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4056.412795] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4060.318327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4068.199746] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4068.557518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4079.523576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4079.715381] Lustre: lustre-OST0001: new disk, initializing [ 4079.718673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4079.723604] Lustre: 141848:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4079.797856] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.047480] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4089.441034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4089.453427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4089.551853] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4093.601180] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4093.773340] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4098.498784] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4098.662462] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4105.746347] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff895588319800.ost_server_uuid 50 [ 4107.686483] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff895588319800.ost_server_uuid in FULL state after 0 sec [ 4119.406358] Lustre: server umount lustre-OST0001 complete [ 4120.583935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4120.588282] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.592357] LustreError: Skipped 1 previous similar message [ 4120.607480] LustreError: 140274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.622294] LustreError: 140274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4125.670617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4125.685110] Lustre: Skipped 1 previous similar message [ 4125.689316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4125.705790] Lustre: Skipped 2 previous similar messages [ 4129.978251] Lustre: server umount lustre-OST0000 complete [ 4133.856963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4133.868853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.884921] Lustre: Skipped 1 previous similar message [ 4133.890299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.571202] Lustre: server umount lustre-MDT0000 complete [ 4141.029135] LustreError: 138437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.057819] LustreError: 138437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4144.064763] LustreError: 140296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824204 with bad export cookie 2918466673556613243 [ 4144.077171] LustreError: 140296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4144.090590] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4144.418769] Lustre: server umount lustre-MDT0001 complete [ 4164.219258] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4167.719923] Key type lgssc unregistered [ 4168.241522] LNet: 144851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4168.253523] LNetError: 144851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4168.294654] LNet: Removed LNI 192.168.204.114@tcp [ 4169.626346] Key type .llcrypt unregistered [ 4169.636583] Key type ._llcrypt unregistered [ 4218.210077] Key type ._llcrypt registered [ 4218.213214] Key type .llcrypt registered [ 4218.365915] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 4236.401167] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4237.484906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4237.494438] alg: No test for adler32 (adler32-zlib) [ 4238.744563] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 4239.099656] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4240.767180] Key type lgssc registered [ 4241.924700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4279.014257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4279.074051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.316838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4280.346713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4280.410551] Lustre: lustre-MDT0000: new disk, initializing [ 4280.516616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.540617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4284.514756] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4295.556410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4295.649761] Lustre: 149007:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4295.683939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4295.686583] Lustre: Skipped 1 previous similar message [ 4295.732171] Lustre: lustre-MDT0001: new disk, initializing [ 4295.814434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4295.848047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4295.858379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4301.114496] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4311.393744] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4317.094271] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4324.197917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.495603] Lustre: lustre-OST0000: new disk, initializing [ 4324.500437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4324.515926] Lustre: 150977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4324.644049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.557316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4330.574387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4330.690171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4330.783220] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4340.048263] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4344.070455] Lustre: Failing over lustre-OST0000 [ 4344.176976] Lustre: server umount lustre-OST0000 complete [ 4361.859575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.192062] Lustre: 147234:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824406/real 1786824406] req@000000001b303e30 x1873621076182016/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786824422 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4362.212738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.237572] Lustre: Skipped 1 previous similar message [ 4362.252456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.285507] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4363.300275] Lustre: 147233:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824406/real 1786824406] req@00000000263df96e x1873621076182144/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786824422 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4363.336973] Lustre: 147233:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4367.254103] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4367.395935] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4367.398124] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4368.351852] Lustre: 147232:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824411/real 1786824411] req@000000006a42e64e x1873621076183296/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786824427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4368.379475] Lustre: 147232:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4368.748560] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4372.258988] Lustre: Failing over lustre-MDT0000 [ 4372.456087] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.468375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4372.582866] Lustre: server umount lustre-MDT0000 complete [ 4373.540566] Lustre: 147232:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824416/real 1786824416] req@00000000bbde75d8 x1873621076184320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786824432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4373.583964] Lustre: 147232:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4377.503348] LustreError: 149015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.523834] LustreError: 149015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4382.612390] LustreError: 149014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.630994] LustreError: 149014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4387.735625] LustreError: 149016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.760865] LustreError: 149016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4391.425307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.606771] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.810518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.852610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4392.859456] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4396.129733] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4397.036043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4397.040833] Lustre: Skipped 1 previous similar message [ 4397.051946] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4397.121426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4407.265642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.268633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.292316] Lustre: Skipped 3 previous similar messages [ 4407.315195] Lustre: Skipped 1 previous similar message [ 4410.210419] Lustre: server umount lustre-OST0000 complete [ 4417.508771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.526570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4417.538169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4422.635709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4422.655868] Lustre: Skipped 2 previous similar messages [ 4427.746579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.753187] Lustre: Skipped 1 previous similar message [ 4428.257105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4428.510971] Lustre: server umount lustre-MDT0000 complete [ 4431.818945] LustreError: 149001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824491 with bad export cookie 11329982323397946632 [ 4431.823994] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.831070] LustreError: 149001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4432.239532] Lustre: server umount lustre-MDT0001 complete [ 4443.334889] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4446.435244] Key type lgssc unregistered [ 4446.767353] LNet: 154634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4446.783298] LNetError: 154634:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4446.801727] LNet: Removed LNI 192.168.204.114@tcp [ 4448.104762] Key type .llcrypt unregistered [ 4448.113754] Key type ._llcrypt unregistered [ 4486.833657] Key type ._llcrypt registered [ 4486.836265] Key type .llcrypt registered [ 4486.949207] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4488.488050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4488.532907] alg: No test for adler32 (adler32-zlib) [ 4489.687693] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 4490.127864] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4491.895246] Key type lgssc registered [ 4492.804105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4501.417365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4501.436709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.147331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.495448] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4513.248707] LustreError: 156160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.301990] LustreError: 156160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4518.376207] LustreError: 156159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.639445] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4523.495816] LustreError: 156160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.215592] LustreError: 156188:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4524.230434] LustreError: 156188:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4524.250161] LustreError: 156188:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4524.832601] Lustre: server umount lustre-MDT0000 complete [ 4541.323424] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 4551.908646] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4593.219852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.574545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4593.626843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4593.759995] Lustre: lustre-MDT0000: new disk, initializing [ 4593.893585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.927096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4600.414376] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4611.703619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.849380] Lustre: 159492:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4611.923792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4611.933408] Lustre: Skipped 1 previous similar message [ 4612.046881] Lustre: lustre-MDT0001: new disk, initializing [ 4612.224578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4612.274197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4612.285380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4617.221153] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4626.928279] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4633.259903] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4640.885479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4641.203446] Lustre: lustre-OST0000: new disk, initializing [ 4641.210712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4641.215878] Lustre: 161462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4641.360415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.614319] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4649.539454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4649.547169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4649.659408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4656.719070] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4661.354483] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4663.442267] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:12:01 (1786824721) [ 4936.673555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.674871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4936.696682] Lustre: Skipped 1 previous similar message [ 4936.717516] Lustre: Skipped 2 previous similar messages [ 4941.788589] Lustre: server umount lustre-MDT0000 complete [ 4941.796268] LustreError: 159500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.806571] LustreError: 159500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4945.444967] LustreError: 159484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825005 with bad export cookie 2572039783374009291 [ 4945.449428] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.458397] LustreError: 159484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4946.912102] LustreError: 163075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.918656] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.948840] Lustre: Skipped 1 previous similar message [ 4946.952071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4950.944357] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4951.766595] Lustre: server umount lustre-MDT0001 complete [ 4961.744030] Lustre: server umount lustre-OST0000 complete [ 4972.180918] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 4984.260294] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5020.506524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.733865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5020.762846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5020.870597] Lustre: lustre-MDT0000: new disk, initializing [ 5020.959599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.984391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5025.617991] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5036.528588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.663362] Lustre: 166127:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5036.723879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5036.729316] Lustre: Skipped 1 previous similar message [ 5036.836791] Lustre: lustre-MDT0001: new disk, initializing [ 5036.932315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.972748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5036.982361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5041.402297] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5050.257869] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5055.311731] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5062.133520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5062.495935] Lustre: lustre-OST0000: new disk, initializing [ 5062.500061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5062.504477] Lustre: 168095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5062.595802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.370457] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5070.402588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5070.420582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5070.537730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5080.517465] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.313857] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5085.498482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5090.919838] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5091.137365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5096.418414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5096.434510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.444839] LustreError: Skipped 1 previous similar message [ 5096.463819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5096.470936] Lustre: Skipped 1 previous similar message [ 5099.307510] Lustre: server umount lustre-OST0000 complete [ 5103.587400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.607221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.626556] Lustre: Skipped 1 previous similar message [ 5103.641380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.979049] Lustre: server umount lustre-MDT0000 complete [ 5111.776494] LustreError: 166936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.790125] LustreError: 166936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5114.114201] LustreError: 166117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825174 with bad export cookie 2572039783374168121 [ 5114.117574] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.140131] LustreError: 166117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5114.610442] Lustre: server umount lustre-MDT0001 complete [ 5125.111465] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:19:42 (1786825182) [ 5146.775074] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 5158.812205] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5195.485553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.712258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5195.733309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5195.819875] Lustre: lustre-MDT0000: new disk, initializing [ 5195.907421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5195.922219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5201.156556] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5212.985630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.127843] Lustre: 172854:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5213.382877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5218.039403] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5227.396234] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5232.932718] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5240.505777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5240.759350] Lustre: lustre-OST0000: new disk, initializing [ 5240.762148] Lustre: Skipped 1 previous similar message [ 5240.765823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5240.773600] Lustre: Skipped 2 previous similar messages [ 5240.778978] Lustre: 174831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5246.996661] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5249.619143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5249.628831] Lustre: Skipped 1 previous similar message [ 5249.634468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5249.796809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5257.770324] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5265.377371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.388102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5265.412801] Lustre: Skipped 1 previous similar message [ 5265.421110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5265.438516] Lustre: Skipped 3 previous similar messages [ 5279.199133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5279.380460] Lustre: server umount lustre-OST0000 complete [ 5285.349164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5285.358161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.368189] Lustre: Skipped 1 previous similar message [ 5285.373135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.377290] Lustre: Skipped 4 previous similar messages [ 5290.977662] LustreError: 172859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.997892] LustreError: 172859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5291.407868] Lustre: server umount lustre-MDT0000 complete [ 5295.372277] LustreError: 172846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825355 with bad export cookie 2572039783374169003 [ 5295.373615] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5295.380677] LustreError: 172846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5295.739716] Lustre: server umount lustre-MDT0001 complete [ 5304.126670] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:22:42 (1786825362) [ 5321.650825] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 5331.139215] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5368.653391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.933706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5368.987383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5369.072421] Lustre: lustre-MDT0000: new disk, initializing [ 5369.167387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.174055] Lustre: Skipped 2 previous similar messages [ 5369.202924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5373.970633] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5385.350245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.496842] Lustre: 179158:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5385.580621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5385.588512] Lustre: Skipped 1 previous similar message [ 5385.792811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5390.251511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5401.900102] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5408.205647] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.579715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.978560] Lustre: 181136:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5422.786387] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5424.697243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5424.831364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5432.254962] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5440.484469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5440.484882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.527601] Lustre: Skipped 2 previous similar messages [ 5445.021355] Lustre: server umount lustre-OST0000 complete [ 5450.733316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.739309] Lustre: Skipped 3 previous similar messages [ 5455.797948] Lustre: server umount lustre-MDT0000 complete [ 5455.846535] LustreError: 180725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.867986] LustreError: 180725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5460.749380] LustreError: 179151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825520 with bad export cookie 2572039783374170088 [ 5460.767173] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.493201] Lustre: server umount lustre-MDT0001 complete [ 5474.761557] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5477.568291] Key type lgssc unregistered [ 5477.848387] LNet: 182993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5477.861529] LNetError: 182993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5477.888515] LNet: Removed LNI 192.168.204.114@tcp [ 5478.795954] Key type .llcrypt unregistered [ 5478.801796] Key type ._llcrypt unregistered [ 5501.962413] Key type ._llcrypt registered [ 5501.964458] Key type .llcrypt registered [ 5502.065876] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5503.086367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5503.121189] alg: No test for adler32 (adler32-zlib) [ 5504.219438] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 5504.521936] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5506.239185] Key type lgssc registered [ 5507.248446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5515.455238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5515.495233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5517.004060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5523.036161] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5526.503177] LustreError: 184411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.540563] LustreError: 184411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5531.625845] LustreError: 184412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.227213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.941772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5541.535800] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5551.152101] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5557.301234] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5564.746379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.111065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.343422] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5581.600442] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.697708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5606.377457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5606.391854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.410506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.398629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.403331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.406988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5607.420123] Lustre: Skipped 1 previous similar message [ 5610.423516] Lustre: server umount lustre-OST0000 complete [ 5617.633102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.634288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5617.666996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5621.657863] Lustre: server umount lustre-MDT0000 complete [ 5622.756889] LustreError: 186918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.777227] LustreError: 186918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5626.813889] LustreError: 184392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825686 with bad export cookie 15389597885925257552 [ 5626.834496] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.844851] LustreError: 184392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5627.558168] Lustre: server umount lustre-MDT0001 complete [ 5639.384092] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 5642.192089] Key type lgssc unregistered [ 5642.544742] LNet: 187971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.552460] LNetError: 187971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5643.623564] LNet: Removed LNI 192.168.204.114@tcp [ 5644.459149] Key type .llcrypt unregistered [ 5644.461268] Key type ._llcrypt unregistered [ 5667.942801] Key type ._llcrypt registered [ 5667.947125] Key type .llcrypt registered [ 5668.081811] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 5669.216676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5669.287050] alg: No test for adler32 (adler32-zlib) [ 5670.389563] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 5670.652055] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 5672.383305] Key type lgssc registered [ 5673.480435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5681.525754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5681.542943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.036274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.900470] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5693.427598] LustreError: 189392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.468849] LustreError: 189392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5697.481969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.889733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5703.131723] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5711.053590] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5717.893301] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5725.743694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.105458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5728.205200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5732.543464] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5740.507518] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5747.680604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5747.686457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.715053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5747.734259] Lustre: Skipped 1 previous similar message [ 5747.749314] Lustre: Skipped 1 previous similar message [ 5752.802218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.808039] Lustre: Skipped 1 previous similar message [ 5753.725721] Lustre: server umount lustre-OST0000 complete [ 5762.751343] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8955834a3000.ost_server_uuid 50 [ 5764.661446] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5779.939163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5779.946987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.979394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.983687] Lustre: Skipped 1 previous similar message [ 5782.497550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5782.504625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.490596] Lustre: server umount lustre-MDT0000 complete [ 5787.619699] LustreError: 190374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.649949] LustreError: 190374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5789.416614] LustreError: 189372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825849 with bad export cookie 2080840829805392860 [ 5789.434670] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.440825] LustreError: 189372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5790.345177] Lustre: server umount lustre-MDT0001 complete [ 5804.157651] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:31:02 (1786825862) [ 5810.677663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.162484] LustreError: 193311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.341082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.170940] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5816.802308] LustreError: 193312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.922360] LustreError: 193311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5823.797919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.227926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5830.171250] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5838.505302] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5844.443153] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5851.628520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5852.038421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.122805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5858.641777] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5867.554508] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5874.566510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5874.772408] Lustre: lustre-OST0001: new disk, initializing [ 5874.778626] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5874.800630] Lustre: 196155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5874.906493] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5881.552037] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5884.531768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5884.540082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5884.692274] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5892.224382] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5895.144044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.170595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.171610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.186264] Lustre: Skipped 2 previous similar messages [ 5900.829238] Lustre: server umount lustre-OST0000 complete [ 5903.269545] LustreError: 195861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.068356] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5908.281566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5912.337309] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5912.454794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5920.736962] LustreError: 195132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.756793] LustreError: 195132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5930.980377] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5930.984797] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.994368] LustreError: Skipped 1 previous similar message [ 5930.999030] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5931.018894] Lustre: Skipped 2 previous similar messages [ 5931.054918] Lustre: Skipped 4 previous similar messages [ 5933.138554] Lustre: server umount lustre-OST0001 complete [ 5941.218553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.233657] Lustre: Skipped 1 previous similar message [ 5943.869049] Lustre: server umount lustre-MDT0000 complete [ 5947.756270] LustreError: 196153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826007 with bad export cookie 2080840829805393931 [ 5947.760854] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.784988] LustreError: 196153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5948.476880] Lustre: server umount lustre-MDT0001 complete [ 5976.447718] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:33:53 (1786826033) [ 5984.237666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.541756] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5984.580968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5984.788304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.393395] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 5998.751223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.884926] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5998.933842] Lustre: 199591:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6003.922961] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6011.806043] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6017.315813] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6024.303845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.546360] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6032.027264] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6033.088885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6039.918667] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6047.998038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6048.041388] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6054.439951] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6058.185839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6062.356867] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6068.166307] Lustre: lustre-OST0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 6068.176471] Lustre: Skipped 2 previous similar messages [ 6068.709754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.723132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.265822] Lustre: server umount lustre-OST0000 complete [ 6073.250683] LustreError: 201444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.290842] LustreError: 201444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6089.186277] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6092.952793] Lustre: server umount lustre-OST0001 complete [ 6102.888623] Lustre: server umount lustre-MDT0000 complete [ 6106.556205] LustreError: 199586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826166 with bad export cookie 2080840829805395338 [ 6106.572911] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6106.578613] LustreError: 199586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6106.883982] Lustre: server umount lustre-MDT0001 complete [ 6131.813851] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:36:29 (1786826189) [ 6148.535620] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6159.104791] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6198.554100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.755418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6198.793335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6198.896867] Lustre: lustre-MDT0000: new disk, initializing [ 6198.979110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.984532] Lustre: Skipped 3 previous similar messages [ 6198.997363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6203.782485] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6215.405773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.568753] Lustre: 207688:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6215.620073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6215.624186] Lustre: Skipped 1 previous similar message [ 6215.714867] Lustre: lustre-MDT0001: new disk, initializing [ 6215.885155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6215.911449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6220.809316] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6232.475725] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6238.427201] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6245.969805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6246.252724] Lustre: lustre-OST0000: new disk, initializing [ 6246.256712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6246.265305] Lustre: 209657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6253.230444] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6256.723671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6256.750369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6256.881867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6262.887233] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6267.932510] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6268.182350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6273.159556] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6273.378115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6277.602302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6277.608686] LustreError: Skipped 1 previous similar message [ 6277.611051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.619654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6277.619664] Lustre: Skipped 6 previous similar messages [ 6277.643847] Lustre: Skipped 6 previous similar messages [ 6281.576580] Lustre: server umount lustre-OST0000 complete [ 6300.127124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6300.331596] Lustre: server umount lustre-MDT0000 complete [ 6302.189479] LustreError: 207702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.228841] LustreError: 207702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6303.858295] LustreError: 208494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826363 with bad export cookie 2080840829805397123 [ 6303.859371] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6303.869854] LustreError: 208494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6304.228206] Lustre: server umount lustre-MDT0001 complete [ 6313.598348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.866978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6328.229031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.696027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.700091] Lustre: Skipped 3 previous similar messages [ 6334.183205] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6342.566811] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6348.983351] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6356.512650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.812831] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6373.048599] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6379.312067] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6379.648701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6385.248252] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6385.507114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6388.742357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.750241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6388.764560] Lustre: Skipped 2 previous similar messages [ 6388.777391] LustreError: Skipped 1 previous similar message [ 6388.784727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6388.819598] Lustre: Skipped 8 previous similar messages [ 6394.349524] Lustre: server umount lustre-OST0000 complete [ 6401.610161] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6407.000472] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6438.730337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6443.324700] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.342093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.345669] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6443.375872] Lustre: Skipped 1 previous similar message [ 6446.907618] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6455.767367] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6457.672791] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6463.971692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6463.980919] LustreError: Skipped 1 previous similar message [ 6468.219313] Lustre: server umount lustre-OST0000 complete [ 6479.034678] Lustre: server umount lustre-MDT0000 complete [ 6479.331669] LustreError: 211875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.354308] LustreError: 211875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6483.488223] LustreError: 211855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826543 with bad export cookie 2080840829805398005 [ 6483.497690] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.506013] LustreError: 211855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6483.748108] Lustre: server umount lustre-MDT0001 complete [ 6492.429430] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:42:30 (1786826550) [ 6498.800284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.550266] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6512.648180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.901744] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6529.498954] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6536.508515] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6544.853455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6552.994440] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6562.732053] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6568.331318] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6570.740779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6576.529614] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6576.834321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6584.575912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.796375] Lustre: lustre-OST0001: new disk, initializing [ 6584.800825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6584.806364] Lustre: 221233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6584.875831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6584.884110] Lustre: Skipped 5 previous similar messages [ 6592.119564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6592.138783] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6592.291171] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6592.841295] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6604.561339] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6611.967816] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6612.144532] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6617.028474] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6617.191201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6623.203779] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6623.206613] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.219029] LustreError: Skipped 1 previous similar message [ 6623.221769] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6623.237935] Lustre: Skipped 5 previous similar messages [ 6623.261756] Lustre: Skipped 7 previous similar messages [ 6625.712519] Lustre: server umount lustre-OST0001 complete [ 6633.369748] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6633.654470] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6639.390207] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6639.690755] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6670.455867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6675.952258] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6675.970842] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6675.971826] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6676.009291] Lustre: Skipped 1 previous similar message [ 6677.630862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6685.750814] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6687.782904] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6696.151415] Lustre: server umount lustre-OST0001 complete [ 6729.183460] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6729.402375] Lustre: server umount lustre-OST0000 complete [ 6748.127160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6748.429118] Lustre: server umount lustre-MDT0000 complete [ 6752.747105] LustreError: 217847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.774954] LustreError: 217847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 6752.826093] LustreError: 217829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826812 with bad export cookie 2080840829805398971 [ 6752.839619] LustreError: 217829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6752.842296] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6759.260478] Lustre: server umount lustre-MDT0001 complete [ 6787.983800] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:47:25 (1786826845) [ 6796.012613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.558977] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6796.639092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6803.544339] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6815.771329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.000770] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6816.060916] Lustre: 226479:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6821.925352] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6830.907522] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6837.689721] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6844.855606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.171824] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6853.402923] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6861.323759] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6872.042387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.136758] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6881.344382] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6891.381687] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6893.872804] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6899.365333] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6899.630687] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6904.393315] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6904.630225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6911.675933] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8955829c4000.ost_server_uuid 50 [ 6913.716904] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8955829c4000.ost_server_uuid in IDLE state after 0 sec [ 6915.174424] Lustre: Permanently deactivating lustre-OST0001 [ 6915.205622] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6919.374380] Lustre: Permanently reactivating lustre-OST0001 [ 6919.387781] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6919.404235] Lustre: Skipped 2 previous similar messages [ 6924.772976] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6924.773898] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.787049] LustreError: Skipped 3 previous similar messages [ 6924.791058] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6924.812751] Lustre: Skipped 7 previous similar messages [ 6924.837025] Lustre: Skipped 15 previous similar messages [ 6930.264147] Lustre: server umount lustre-OST0001 complete [ 6940.755514] Lustre: server umount lustre-OST0000 complete [ 6951.994306] Lustre: server umount lustre-MDT0000 complete [ 6957.178299] LustreError: 228343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827017 with bad export cookie 2080840829805400259 [ 6957.181990] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6957.189194] LustreError: 228343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6957.644672] Lustre: server umount lustre-MDT0001 complete [ 6985.924803] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:50:44 (1786827044) [ 6995.024395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.593495] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6995.648482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.660148] Lustre: Skipped 2 previous similar messages [ 7001.681731] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7011.226724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.392542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7011.476939] Lustre: 233183:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7016.560763] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7024.536446] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7032.909906] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7042.507346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7042.823849] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7042.886606] Lustre: Permanently deactivating lustre-OST0000 [ 7042.913533] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7049.483271] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7050.768074] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7050.835585] LustreError: 188699:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e7094c4a x1873622577145344/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7056.959578] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7064.403688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.461751] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7070.400702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7071.598728] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7079.283596] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7081.144671] Lustre: Permanently reactivating lustre-OST0000 [ 7081.160464] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7081.165701] Lustre: Skipped 2 previous similar messages [ 7090.271422] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7090.290746] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7090.303668] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7090.369375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7103.067337] Lustre: server umount lustre-OST0001 complete [ 7121.375670] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7121.743141] Lustre: server umount lustre-OST0000 complete [ 7132.424984] Lustre: server umount lustre-MDT0000 complete [ 7136.792035] LustreError: 233173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827196 with bad export cookie 2080840829805401932 [ 7136.814897] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.824380] LustreError: 233173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7137.273109] Lustre: server umount lustre-MDT0001 complete [ 7149.947753] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7152.860906] Key type lgssc unregistered [ 7153.308618] LNet: 238068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7153.327022] LNetError: 238068:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7153.373180] LNet: Removed LNI 192.168.204.114@tcp [ 7154.526391] Key type .llcrypt unregistered [ 7154.529094] Key type ._llcrypt unregistered [ 7179.959564] Key type ._llcrypt registered [ 7179.963510] Key type .llcrypt registered [ 7180.051595] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7181.440726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7181.572941] alg: No test for adler32 (adler32-zlib) [ 7182.775842] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 7183.282429] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7184.993292] Key type lgssc registered [ 7186.260123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7197.722646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7197.748947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7199.122256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.023505] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7209.443272] LustreError: 239536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7209.469235] LustreError: 239536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7212.942886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.065629] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7213.087827] Lustre: 239523:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7213.096602] Lustre: Permanently deactivating lustre-MDT0001 [ 7213.107911] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7213.374783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.985188] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7217.000154] LustreError: 239566:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7217.013385] LustreError: 239566:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7217.040243] LustreError: 239566:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7218.602869] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7226.895783] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7233.165993] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7240.955996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7241.345782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.398557] LustreError: 241379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7244.415965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7247.997392] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7249.906884] LustreError: 241379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.013443] LustreError: 241379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.848503] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7258.102221] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7263.207054] LustreError: 241545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.225727] LustreError: 241545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7265.294940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.590128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7270.917606] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7270.933765] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7273.311499] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7282.379201] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7285.959740] Lustre: Permanently reactivating lustre-MDT0001 [ 7285.999810] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7286.008623] Lustre: Skipped 2 previous similar messages [ 7289.186039] Lustre: lustre-MDT0001: Client f37a7dbd-a8da-4a57-a6e2-87ac3044a492 (at 192.168.204.14@tcp) reconnecting [ 7295.990640] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7297.690887] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7303.324261] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7303.572475] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7308.768628] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7308.927226] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7311.324266] Lustre: Permanently deactivating lustre-MDT0001 [ 7311.337821] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7311.348385] Lustre: Skipped 2 previous similar messages [ 7317.184801] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7331.297314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7331.319402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.355044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.338195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.351157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.363044] Lustre: Skipped 2 previous similar messages [ 7332.373840] Lustre: Skipped 3 previous similar messages [ 7335.410832] Lustre: server umount lustre-MDT0000 complete [ 7337.440302] LustreError: 239535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.468804] LustreError: 239535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7340.522971] LustreError: 243085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827400 with bad export cookie 5350973380356412169 [ 7340.537226] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.549263] LustreError: 243085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7340.981505] Lustre: server umount lustre-MDT0001 complete [ 7347.985984] Lustre: server umount lustre-OST0000 complete [ 7360.409633] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:56:58 (1786827418) [ 7372.436261] Lustre: server umount lustre-OST0001 complete [ 7380.028933] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7391.017581] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7432.202511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.431605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7432.439363] Lustre: Skipped 2 previous similar messages [ 7432.466092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7432.530316] Lustre: lustre-MDT0000: new disk, initializing [ 7432.594886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.608901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7437.364538] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7450.061894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.217535] Lustre: 247825:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7450.284803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7450.289663] Lustre: Skipped 1 previous similar message [ 7450.377428] Lustre: lustre-MDT0001: new disk, initializing [ 7450.510345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7450.551056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7450.565754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7455.522361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7466.652881] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7472.770990] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7480.612478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7481.008124] Lustre: lustre-OST0000: new disk, initializing [ 7481.014873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7481.024703] Lustre: 249794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7481.148778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.926814] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7490.605496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7490.617909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7490.770915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7499.113328] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7501.923267] LustreError: 247832:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7512.015076] LustreError: 247832:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7512.057708] LustreError: 249802:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7512.139396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.418587] Lustre: lustre-OST0001: new disk, initializing [ 7512.424749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7512.436030] Lustre: 250909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7518.874343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7518.885764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7519.020317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7520.287938] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7522.119690] LustreError: 249802:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7532.762226] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7534.838771] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7539.680209] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.693166] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.701860] LustreError: Skipped 1 previous similar message [ 7539.712473] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7539.731054] Lustre: Skipped 1 previous similar message [ 7539.761989] Lustre: Skipped 1 previous similar message [ 7543.046115] Lustre: server umount lustre-OST0001 complete [ 7544.805150] LustreError: 249777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.829300] LustreError: 249777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7562.720655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7562.728068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7562.740526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7562.744722] Lustre: Skipped 1 previous similar message [ 7566.506822] Lustre: server umount lustre-OST0000 complete [ 7570.419830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7570.427420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.447290] Lustre: Skipped 2 previous similar messages [ 7570.456988] Lustre: Skipped 3 previous similar messages [ 7575.864993] Lustre: server umount lustre-MDT0000 complete [ 7580.186162] LustreError: 247818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827640 with bad export cookie 5350973380356414983 [ 7580.187042] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7580.198952] LustreError: 247818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7586.755034] Lustre: server umount lustre-MDT0001 complete [ 7597.799655] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7600.851957] Key type lgssc unregistered [ 7601.149102] LNet: 252939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7601.154563] LNetError: 252939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7601.171940] LNet: Removed LNI 192.168.204.114@tcp [ 7602.406827] Key type .llcrypt unregistered [ 7602.415899] Key type ._llcrypt unregistered [ 7646.751223] Key type ._llcrypt registered [ 7646.755491] Key type .llcrypt registered [ 7646.949802] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7648.581792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7648.603217] alg: No test for adler32 (adler32-zlib) [ 7649.694934] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 7650.028743] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7651.791434] Key type lgssc registered [ 7653.360265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7662.802505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7662.823477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.162691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7664.178533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7664.331869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.855812] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7680.024382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7680.108774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7680.132359] Lustre: 255161:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7680.382411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7685.872896] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7697.760561] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7707.891291] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7715.998232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.377527] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7716.730500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.225483] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7724.720903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7733.670267] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7791.074298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7791.088704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.108882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7791.591627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7791.598050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.610613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.269402] Lustre: server umount lustre-OST0000 complete [ 7799.285630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7812.764840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7813.168658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.970792] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7829.130539] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7831.539525] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7831.559420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7831.560518] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7872.482600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7872.489556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.492280] LustreError: Skipped 1 previous similar message [ 7872.493641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.522536] Lustre: Skipped 1 previous similar message [ 7875.460291] Lustre: server umount lustre-OST0000 complete [ 7880.161718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7880.172824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.186591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.190779] Lustre: Skipped 1 previous similar message [ 7885.602937] Lustre: server umount lustre-MDT0000 complete [ 7889.686365] LustreError: 255906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827949 with bad export cookie 4437727348640566620 [ 7889.694262] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7889.699770] LustreError: 255906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7890.004121] Lustre: server umount lustre-MDT0001 complete [ 7902.597210] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7905.408566] Key type lgssc unregistered [ 7905.723982] LNet: 260587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7905.735638] LNetError: 260587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7905.752264] LNet: Removed LNI 192.168.204.114@tcp [ 7906.665332] Key type .llcrypt unregistered [ 7906.666973] Key type ._llcrypt unregistered [ 7931.995573] Key type ._llcrypt registered [ 7931.997259] Key type .llcrypt registered [ 7932.128347] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7933.211782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7933.229810] alg: No test for adler32 (adler32-zlib) [ 7934.287283] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 7934.491435] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7936.151326] Key type lgssc registered [ 7937.395404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7945.602759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7945.625890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.242261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.086374] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7957.480611] LustreError: 262006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.500063] LustreError: 262006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7960.615303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.004990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7965.302506] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7972.865547] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7979.009594] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7985.584462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7985.706346] LustreError: 263829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7985.914409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.317396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7992.655511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8000.280088] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8027.104430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8027.113331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.134507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8028.130909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.133097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.010313] Lustre: server umount lustre-OST0000 complete [ 8037.863802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8037.873541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.887933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.894376] Lustre: Skipped 1 previous similar message [ 8040.003342] Lustre: server umount lustre-MDT0000 complete [ 8043.488766] LustreError: 262006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.507118] LustreError: 262006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8044.382307] LustreError: 261987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828104 with bad export cookie 16234715007272489301 [ 8044.390769] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8044.395730] LustreError: 261987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8044.699259] Lustre: server umount lustre-MDT0001 complete [ 8057.730439] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8061.911202] Key type lgssc unregistered [ 8062.285763] LNet: 266315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8062.292826] LNetError: 266315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8063.343989] LNet: Removed LNI 192.168.204.114@tcp [ 8064.672661] Key type .llcrypt unregistered [ 8064.680148] Key type ._llcrypt unregistered [ 8082.983244] Key type ._llcrypt registered [ 8082.986838] Key type .llcrypt registered [ 8083.075425] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8084.197680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8084.342771] alg: No test for adler32 (adler32-zlib) [ 8085.369433] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 8085.627345] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8087.383210] Key type lgssc registered [ 8089.007209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8098.024360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8098.050572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8099.624144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.360556] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8109.027862] LustreError: 267536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.060820] LustreError: 267536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8113.937723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8114.149422] LustreError: 267537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.380703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8119.357875] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8127.787821] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8133.748630] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8141.087816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.595054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.781603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8149.403879] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8158.234262] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8174.562837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.586768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.596403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8174.649619] Lustre: Skipped 1 previous similar message [ 8177.477576] Lustre: server umount lustre-OST0000 complete [ 8184.803025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.808224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.811778] Lustre: Skipped 1 previous similar message [ 8184.819989] Lustre: Skipped 2 previous similar messages [ 8188.932184] Lustre: server umount lustre-MDT0000 complete [ 8189.926875] LustreError: 267536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.943351] LustreError: 267536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8193.719238] LustreError: 267518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828253 with bad export cookie 5295680179107765709 [ 8193.720904] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.737521] LustreError: 267518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8194.168943] Lustre: server umount lustre-MDT0001 complete [ 8205.420587] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8208.400853] Key type lgssc unregistered [ 8208.729248] LNet: 271214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8208.737212] LNetError: 271214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8208.767881] LNet: Removed LNI 192.168.204.114@tcp [ 8209.854955] Key type .llcrypt unregistered [ 8209.857346] Key type ._llcrypt unregistered [ 8235.636905] Key type ._llcrypt registered [ 8235.646208] Key type .llcrypt registered [ 8235.775277] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8237.081960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.199756] alg: No test for adler32 (adler32-zlib) [ 8238.337672] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 8238.674144] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8240.417103] Key type lgssc registered [ 8242.483233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8254.229792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8254.247213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.957311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.795911] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8265.186611] LustreError: 272635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8265.211880] LustreError: 272635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8270.320243] LustreError: 272636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8271.343603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.956663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.578880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8286.344756] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8293.361955] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8301.276112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8301.742954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.026755] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8320.456749] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8325.317275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8351.201053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8351.203948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.215812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.234216] Lustre: Skipped 1 previous similar message [ 8356.326332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8356.329989] Lustre: Skipped 1 previous similar message [ 8356.757972] Lustre: server umount lustre-OST0000 complete [ 8361.440838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.450164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8361.463768] Lustre: Skipped 1 previous similar message [ 8361.489027] Lustre: Skipped 2 previous similar messages [ 8366.563128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8366.575939] Lustre: Skipped 1 previous similar message [ 8367.356331] Lustre: server umount lustre-MDT0000 complete [ 8371.684528] LustreError: 272631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.696358] LustreError: 272631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8372.041777] LustreError: 272616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828432 with bad export cookie 18091400261418970946 [ 8372.042389] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8372.064797] LustreError: 272616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8372.328663] Lustre: server umount lustre-MDT0001 complete [ 8383.434100] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8386.574791] Key type lgssc unregistered [ 8386.854761] LNet: 276945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8386.861848] LNetError: 276945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8387.878159] LNet: Removed LNI 192.168.204.114@tcp [ 8389.010241] Key type .llcrypt unregistered [ 8389.015262] Key type ._llcrypt unregistered [ 8408.798438] Key type ._llcrypt registered [ 8408.800619] Key type .llcrypt registered [ 8408.918056] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8410.243398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8410.382562] alg: No test for adler32 (adler32-zlib) [ 8411.533759] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 8411.728476] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8413.359168] Key type lgssc registered [ 8414.830906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8424.033421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8424.072847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8425.478144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.664134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8435.685649] LustreError: 278181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8435.703947] LustreError: 278181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8439.320205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.904579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8444.883632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8453.310327] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8459.829103] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8467.023286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.382453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.459345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8474.605142] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8482.902234] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8496.098544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8496.108116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.128241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8497.473912] Lustre: server umount lustre-OST0000 complete [ 8505.321920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.324997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.339746] Lustre: Skipped 1 previous similar message [ 8505.359841] Lustre: Skipped 1 previous similar message [ 8507.654323] Lustre: server umount lustre-MDT0000 complete [ 8510.438955] LustreError: 278173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8510.463613] LustreError: 278173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8511.916270] LustreError: 278147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828571 with bad export cookie 8554235742086050617 [ 8511.917801] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8511.923357] LustreError: 278147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8521.461456] Lustre: server umount lustre-MDT0001 complete [ 8535.087252] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8538.028051] Key type lgssc unregistered [ 8538.357856] LNet: 281884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8538.386282] LNetError: 281884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8538.406930] LNet: Removed LNI 192.168.204.114@tcp [ 8539.128208] Key type .llcrypt unregistered [ 8539.133420] Key type ._llcrypt unregistered [ 8551.767555] Key type ._llcrypt registered [ 8551.769876] Key type .llcrypt registered [ 8551.977550] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8569.608950] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8585.816862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8586.652936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8586.715518] alg: No test for adler32 (adler32-zlib) [ 8587.897164] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 8588.131718] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8589.807462] Key type lgssc registered [ 8591.067699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8624.288046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8624.336654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.552489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8625.581428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8625.672307] Lustre: lustre-MDT0000: new disk, initializing [ 8625.765646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.788812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8630.277969] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8640.627446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.718503] Lustre: 285441:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8640.795751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8640.801367] Lustre: Skipped 1 previous similar message [ 8640.830663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8640.893458] Lustre: lustre-MDT0001: new disk, initializing [ 8640.996763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8641.033214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8641.052964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8645.643908] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8655.782094] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8661.691086] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8669.664138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8670.084444] Lustre: lustre-OST0000: new disk, initializing [ 8670.098769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8670.116790] Lustre: 287410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8670.282162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8677.618293] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8679.508841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8679.517018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8679.623897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8687.263183] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8692.437657] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8692.739600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8698.054382] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8698.227808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8705.515784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.525499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8705.539735] Lustre: Skipped 1 previous similar message [ 8705.555876] Lustre: Skipped 1 previous similar message [ 8707.137617] Lustre: server umount lustre-OST0000 complete [ 8712.672350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8712.683755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.696216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8715.759552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.760938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8717.122257] Lustre: server umount lustre-MDT0000 complete [ 8720.413776] LustreError: 285431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828780 with bad export cookie 17054254843041554995 [ 8720.423255] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.424836] LustreError: 285431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8720.777904] Lustre: server umount lustre-MDT0001 complete [ 8728.649749] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:19:47 (1786828787) [ 8734.499307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.004874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8738.602158] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8740.328288] LustreError: 289821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.343269] LustreError: 289821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8744.420115] LustreError: 289822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8746.005917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8746.394916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8750.191792] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8757.380175] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8762.302605] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8769.242938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8769.523704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8776.380372] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8784.510509] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8795.112886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8795.113561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.128823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8795.128830] Lustre: Skipped 1 previous similar message [ 8795.155580] Lustre: Skipped 1 previous similar message [ 8799.722793] Lustre: server umount lustre-OST0000 complete [ 8805.355604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8805.365395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8805.377203] Lustre: Skipped 1 previous similar message [ 8805.384519] Lustre: Skipped 1 previous similar message [ 8818.143368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8818.251662] Lustre: server umount lustre-MDT0000 complete [ 8820.704390] LustreError: 292313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.723269] LustreError: 292313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8821.883298] LustreError: 289802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828881 with bad export cookie 17054254843041555870 [ 8821.891553] LustreError: 289802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8821.894274] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8822.152215] Lustre: server umount lustre-MDT0001 complete [ 8832.737390] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8835.466441] Key type lgssc unregistered [ 8835.771969] LNet: 293354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8835.782648] LNetError: 293354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8835.804309] LNet: Removed LNI 192.168.204.114@tcp [ 8836.703592] Key type .llcrypt unregistered [ 8836.706340] Key type ._llcrypt unregistered [ 8862.400084] Key type ._llcrypt registered [ 8862.406882] Key type .llcrypt registered [ 8862.525208] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8876.648344] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8877.465399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8877.532331] alg: No test for adler32 (adler32-zlib) [ 8878.592262] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 8878.769171] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8880.439175] Key type lgssc registered [ 8881.844077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8934.442889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8934.469907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.791989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8935.832833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8935.914551] Lustre: lustre-MDT0000: new disk, initializing [ 8935.978100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.994431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8940.993302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8951.081170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.238589] Lustre: 297186:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8951.326733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8951.344493] Lustre: Skipped 1 previous similar message [ 8951.433531] Lustre: lustre-MDT0001: new disk, initializing [ 8951.555964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8951.591300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8951.601249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8955.754501] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8964.925917] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8970.256202] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8976.884787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8977.171624] Lustre: lustre-OST2710: new disk, initializing [ 8977.180986] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8977.191436] Lustre: 299152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8977.279490] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8983.152773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8983.167457] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8983.298525] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8983.308785] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8992.587811] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8999.911377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.210723] Lustre: lustre-OST03e8: new disk, initializing [ 9000.216573] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9000.223780] Lustre: 300215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9000.337300] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9005.860462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9005.887304] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9006.135123] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9010.124816] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9020.047422] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9027.797704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9028.059156] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9032.681298] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9032.990321] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9038.285755] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9038.574098] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9043.968855] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9044.157084] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9049.057303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9049.074153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.087703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9051.239233] Lustre: server umount lustre-MDT0000 complete [ 9052.152584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.745697] LustreError: 297176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829114 with bad export cookie 2056070009805543574 [ 9054.748412] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9054.754840] LustreError: 297176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9055.194172] Lustre: server umount lustre-MDT0001 complete [ 9068.214706] Lustre: server umount lustre-OST2710 complete [ 9081.802591] Lustre: server umount lustre-OST03e8 complete [ 9102.109367] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9112.072796] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9153.820823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.132215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9154.167908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9154.243414] Lustre: lustre-MDT0000: new disk, initializing [ 9154.358994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.382229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9158.774949] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9170.716928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.838980] Lustre: 305464:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9170.885915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9170.889491] Lustre: Skipped 1 previous similar message [ 9170.954579] Lustre: lustre-MDT0001: new disk, initializing [ 9171.072664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.131972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9171.149180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9176.317475] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9186.607465] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9194.747773] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9203.477663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9203.714296] Lustre: lustre-OST0000: new disk, initializing [ 9203.717426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9203.721961] Lustre: 307438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9203.802350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.997426] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9212.539473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9212.551906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9212.680958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9223.837923] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9229.306225] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9229.531751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9234.262889] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9234.485122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9238.497267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.529394] Lustre: Skipped 2 previous similar messages [ 9238.542448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9238.555971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.579807] Lustre: Skipped 1 previous similar message [ 9242.742630] Lustre: server umount lustre-OST0000 complete [ 9247.712307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9247.724592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9247.733329] Lustre: Skipped 1 previous similar message [ 9247.744351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9253.858856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9253.867395] Lustre: Skipped 3 previous similar messages [ 9258.980972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9261.023144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9261.252906] Lustre: server umount lustre-MDT0000 complete [ 9264.105983] LustreError: 305476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9264.123292] LustreError: 305476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9265.395455] LustreError: 305457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829325 with bad export cookie 2056070009805545254 [ 9265.403898] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9265.410307] LustreError: 305457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9265.922893] Lustre: server umount lustre-MDT0001 complete [ 9274.052403] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:28:52 (1786829332) [ 9275.461978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9277.184445] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:28:55 (1786829335) [ 9280.716753] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9308.337098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.620503] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9308.639330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9308.848343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.858669] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9322.850399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9322.916860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9322.942768] Lustre: 311296:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9323.057563] LustreError: 311307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.863719] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9351.125448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.548806] Lustre: Denying initial registration attempt from nid 192.168.204.114@tcp, specified as failover [ 9351.562469] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9351.580636] LustreError: 312751:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9351.588901] LustreError: 312751:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9351.597227] LustreError: 312751:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9351.711366] Lustre: server umount lustre-OST0000 complete [ 9351.716054] LustreError: 312751:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9364.451549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.452914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.477045] Lustre: Skipped 2 previous similar messages [ 9364.495159] Lustre: Skipped 2 previous similar messages [ 9366.044988] Lustre: server umount lustre-MDT0000 complete [ 9369.570314] LustreError: 312570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.603136] LustreError: 312570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9370.371681] LustreError: 311289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829430 with bad export cookie 2056070009805546136 [ 9370.379044] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9370.386452] LustreError: 311289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9370.953826] Lustre: server umount lustre-MDT0001 complete [ 9382.413178] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9385.010977] Key type lgssc unregistered [ 9385.304740] LNet: 313871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9385.310964] LNetError: 313871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9385.327836] LNet: Removed LNI 192.168.204.114@tcp [ 9386.167844] Key type .llcrypt unregistered [ 9386.170201] Key type ._llcrypt unregistered [ 9398.888376] Key type ._llcrypt registered [ 9398.894607] Key type .llcrypt registered [ 9399.052564] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9400.187974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9400.364306] alg: No test for adler32 (adler32-zlib) [ 9401.616470] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 9401.876953] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9403.529402] Key type lgssc registered [ 9404.786776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9456.931240] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9467.513670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9467.531708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.955821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9468.981252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9469.276554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9475.181164] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9484.256428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.392742] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9484.436748] Lustre: 316639:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9484.720867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.814926] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9498.121536] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9504.297777] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9524.392881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9524.614582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9524.870181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.225676] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9540.535965] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9543.144719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9543.164511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9543.194369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9544.899613] Lustre: server umount lustre-OST0000 complete [ 9551.330946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9551.340978] LustreError: Skipped 1 previous similar message [ 9551.344249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9551.362076] Lustre: Skipped 1 previous similar message [ 9551.367287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9551.373903] Lustre: Skipped 1 previous similar message [ 9553.890389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.895450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9555.276312] Lustre: server umount lustre-MDT0000 complete [ 9559.899252] LustreError: 316630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829619 with bad export cookie 1055583311410432988 [ 9559.899865] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9559.913668] LustreError: 316630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9560.418141] Lustre: server umount lustre-MDT0001 complete [ 9572.774335] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9575.997733] Key type lgssc unregistered [ 9576.261802] LNet: 320215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9576.276945] LNetError: 320215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9576.308739] LNet: Removed LNI 192.168.204.114@tcp [ 9577.371805] Key type .llcrypt unregistered [ 9577.378073] Key type ._llcrypt unregistered [ 9601.953900] Key type ._llcrypt registered [ 9601.956706] Key type .llcrypt registered [ 9602.041328] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9603.101104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9603.138662] alg: No test for adler32 (adler32-zlib) [ 9604.193801] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 9604.430973] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9606.138933] Key type lgssc registered [ 9607.200678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9614.747420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9614.794464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.291819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9620.566873] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9626.599073] LustreError: 321631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.614660] LustreError: 321631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9629.302796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9629.728673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9634.092635] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9642.028821] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9647.628703] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9657.003029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.406573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.136478] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9673.047693] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9684.964258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9684.970439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9684.989573] Lustre: Skipped 1 previous similar message [ 9685.005647] Lustre: Skipped 1 previous similar message [ 9685.922865] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 9688.534966] Lustre: server umount lustre-MDT0000 complete [ 9690.090774] LustreError: 322639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9690.124741] LustreError: 322639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9692.654950] LustreError: 322365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829752 with bad export cookie 848843988282138298 [ 9692.656840] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9692.664259] LustreError: 322365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9695.214936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9695.216185] LustreError: 321627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.238872] Lustre: Skipped 1 previous similar message [ 9695.244096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9695.261119] LustreError: 321627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9697.250529] LustreError: 322639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9697.250767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9697.264332] LustreError: 322639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9697.275290] Lustre: Skipped 1 previous similar message [ 9699.081584] Lustre: server umount lustre-MDT0001 complete [ 9705.993438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9714.701645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9717.751455] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc7b31c27487c5e [ 9717.759996] Lustre: MGC192.168.204.114@tcp: Connection restored to 0@lo (at 0@lo) [ 9718.289491] LustreError: 323464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.360279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.425814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9723.267801] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9727.596070] LustreError: 323465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.597274] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9727.646644] LustreError: 323465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9734.516345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.773437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9740.275147] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9741.568736] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9749.212654] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9750.885670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9756.833515] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9758.598073] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9764.323389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9764.333740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9764.343098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9765.344931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9769.652133] Lustre: server umount lustre-OST0000 complete [ 9775.586476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9775.599898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9775.616800] Lustre: Skipped 1 previous similar message [ 9775.621801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9775.632240] Lustre: Skipped 2 previous similar messages [ 9779.481498] Lustre: server umount lustre-MDT0000 complete [ 9781.218993] LustreError: 325231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9781.267246] LustreError: 325231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9783.233292] LustreError: 321611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829843 with bad export cookie 848843988282145886 [ 9783.234965] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9783.248702] LustreError: 321611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9783.500207] Lustre: server umount lustre-MDT0001 complete [ 9797.164304] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9801.501849] Key type lgssc unregistered [ 9801.856501] LNet: 327901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9801.867732] LNetError: 327901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9801.894626] LNet: Removed LNI 192.168.204.114@tcp [ 9803.066469] Key type .llcrypt unregistered [ 9803.068643] Key type ._llcrypt unregistered [ 9827.174052] Key type ._llcrypt registered [ 9827.179480] Key type .llcrypt registered [ 9827.337405] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9828.875021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9829.123048] alg: No test for adler32 (adler32-zlib) [ 9830.212713] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [ 9830.540978] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9832.343255] Key type lgssc registered [ 9833.414620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9841.770457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9841.785716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.283270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9848.511907] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9853.430748] LustreError: 329322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.444980] LustreError: 329322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9856.010029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9856.270950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9860.050494] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9867.174982] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9874.623657] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9881.578813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.580964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9881.608443] Lustre: Skipped 1 previous similar message [ 9881.616119] Lustre: Skipped 1 previous similar message [ 9885.891913] Lustre: server umount lustre-MDT0000 complete [ 9886.695704] LustreError: 330381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.719640] LustreError: 330381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9889.804177] LustreError: 329301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829949 with bad export cookie 17620228667248793206 [ 9889.810475] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9889.824614] LustreError: 329301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9890.258581] Lustre: server umount lustre-MDT0001 complete [ 9906.316085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.628134] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9906.670471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9906.970226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.109921] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9922.759714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9922.931269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9923.007782] Lustre: 331910:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9923.174336] LustreError: 331920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.459938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9927.563080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9935.930152] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9941.921162] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9951.401163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9951.552363] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9951.564804] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9951.570615] LustreError: 333794:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9951.575556] LustreError: 333794:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9951.578938] LustreError: 333794:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9951.640437] Lustre: server umount lustre-OST0000 complete [ 9951.648181] LustreError: 333794:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9961.833741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9962.074269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9962.252974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9968.289539] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9971.907652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9976.050327] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9983.824908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9983.908534] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9983.976850] Lustre: lustre-OST0001: new disk, initializing [ 9983.981162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9983.986712] Lustre: 335336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9984.079579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9990.768362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9990.784213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9990.869864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9990.898224] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9998.378799] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10001.385249] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10001.387000] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.392174] LustreError: Skipped 1 previous similar message [10001.395247] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10001.404478] Lustre: Skipped 1 previous similar message [10001.432520] Lustre: Skipped 1 previous similar message [10006.499768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10006.514228] Lustre: Skipped 1 previous similar message [10007.754227] Lustre: server umount lustre-OST0001 complete [10011.624627] LustreError: 334259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.643401] LustreError: 334259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10012.641890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10012.653634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10012.667305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10016.747384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10016.747915] LustreError: 336341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10016.752981] Lustre: Skipped 2 previous similar messages [10016.767469] LustreError: 336341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10018.539978] Lustre: server umount lustre-OST0000 complete [10025.952642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10025.959587] LustreError: Skipped 1 previous similar message [10025.964199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.983628] Lustre: Skipped 1 previous similar message [10025.997418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.215134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10037.464288] Lustre: server umount lustre-MDT0000 complete [10041.316420] LustreError: 331915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10041.332931] LustreError: 331915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10041.667517] LustreError: 331901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830101 with bad export cookie 17620228667248793759 [10041.684235] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10042.066754] Lustre: server umount lustre-MDT0001 complete [10052.624702] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [10055.312496] Key type lgssc unregistered [10055.635607] LNet: 337295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10055.643330] LNetError: 337295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10055.652514] LNet: Removed LNI 192.168.204.114@tcp [10056.487753] Key type .llcrypt unregistered [10056.489541] Key type ._llcrypt unregistered [10127.215705] Key type ._llcrypt registered [10127.217564] Key type .llcrypt registered [10127.366968] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10142.180336] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10143.882931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10143.987143] alg: No test for adler32 (adler32-zlib) [10145.271848] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [10145.617825] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [10147.407356] Key type lgssc registered [10148.466803] Lustre: Echo OBD driver; http://www.lustre.org/ [10184.700068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10184.719943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.971162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10186.017057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10186.104677] Lustre: lustre-MDT0000: new disk, initializing [10186.188968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10186.211533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10190.278476] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10200.980809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10201.129702] Lustre: 342234:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10201.181734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10201.186551] Lustre: Skipped 1 previous similar message [10201.262512] Lustre: lustre-MDT0001: new disk, initializing [10201.348150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10201.386841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10201.401866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10205.980826] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10215.651952] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10221.310620] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10229.592323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.812263] Lustre: lustre-OST0000: new disk, initializing [10229.815966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10229.820262] Lustre: 344203:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10229.896654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10235.556218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10235.564492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10235.768150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10236.550952] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10247.331202] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10252.755842] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10253.018748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10258.071969] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10258.380595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10261.480658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10261.485620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10261.494293] LustreError: Skipped 1 previous similar message [10261.520205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10261.532762] Lustre: Skipped 1 previous similar message [10266.598351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10266.612402] Lustre: Skipped 1 previous similar message [10271.714798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10271.728093] Lustre: Skipped 1 previous similar message [10275.295136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10275.395794] Lustre: server umount lustre-OST0000 complete [10281.953705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10281.977179] Lustre: Skipped 1 previous similar message [10281.989786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10281.994517] Lustre: Skipped 1 previous similar message [10287.072978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10287.081499] Lustre: Skipped 1 previous similar message [10294.239758] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10294.544833] Lustre: server umount lustre-MDT0000 complete [10298.877769] LustreError: 342225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830358 with bad export cookie 3433092305924930401 [10298.879046] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.894790] LustreError: 342225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10299.238384] Lustre: server umount lustre-MDT0001 complete [10308.582579] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:46:06 (1786830366) [10316.352560] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:46:14 (1786830374) [10324.690166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.306897] LustreError: 346960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.320215] LustreError: 346960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10325.404764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.497121] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10330.597163] LustreError: 346961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.721481] LustreError: 346960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.834613] LustreError: 346961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.493415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10343.126907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10348.583548] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10356.241124] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10361.114378] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10368.818287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10369.060565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10374.762530] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10381.524892] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10393.889772] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10395.858277] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10397.919604] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10400.169850] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10402.273875] Lustre: Failing over lustre-MDT0000 [10402.638252] Lustre: server umount lustre-MDT0000 complete [10402.795410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10402.804287] LustreError: 346954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.836673] Lustre: Skipped 2 previous similar messages [10402.867103] LustreError: 346954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10413.025585] LustreError: 346960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.047203] LustreError: 346960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10421.651328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.895671] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10422.097123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10422.138459] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10426.262344] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10426.605286] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10427.369497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10427.395240] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10427.425786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10436.288223] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10438.381642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10442.607382] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10444.673284] Lustre: Failing over lustre-MDT0000 [10445.028693] Lustre: server umount lustre-MDT0000 complete [10446.268760] LustreError: 346956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.303542] LustreError: 346956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10447.846824] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10447.851694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10447.862828] Lustre: Skipped 1 previous similar message [10463.199154] Lustre: 340457:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830507/real 1786830507] req@00000000a08da091 x1873627269133312/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786830523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10463.239696] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10465.871339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.709495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10473.761345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10477.971889] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10478.481344] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10479.099177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10479.103694] Lustre: Skipped 2 previous similar messages [10479.130026] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10479.196170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10489.450697] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10491.204938] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10499.556717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10499.570536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10499.574128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10499.590214] Lustre: Skipped 4 previous similar messages [10499.618905] Lustre: Skipped 4 previous similar messages [10503.515724] Lustre: server umount lustre-MDT0000 complete [10504.677901] LustreError: 346955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10504.688866] LustreError: 346955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [10507.587352] LustreError: 347695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830567 with bad export cookie 3433092305924933117 [10507.592459] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.604737] LustreError: 347695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10507.839382] Lustre: server umount lustre-MDT0001 complete [10523.646264] Lustre: server umount lustre-OST0000 complete [10540.906265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.389599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10545.556747] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10554.186037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.250910] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10566.337062] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10571.326712] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10578.101203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10584.853207] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10586.677558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10592.618658] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10599.908113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10599.920261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.939840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10600.052632] Lustre: server umount lustre-OST0000 complete [10605.540573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10618.335125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10618.592500] Lustre: server umount lustre-MDT0000 complete [10622.229148] LustreError: 353311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830682 with bad export cookie 3433092305924933859 [10622.235687] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10622.255180] LustreError: 353311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10622.495298] Lustre: server umount lustre-MDT0001 complete [10636.542480] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [10640.316748] Key type lgssc unregistered [10640.708580] LNet: 356861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10640.715873] LNetError: 356861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10640.775398] LNet: Removed LNI 192.168.204.114@tcp [10641.601232] Key type .llcrypt unregistered [10641.603827] Key type ._llcrypt unregistered [10669.488996] Key type ._llcrypt registered [10669.493731] Key type .llcrypt registered [10669.577876] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10670.432267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10670.546714] alg: No test for adler32 (adler32-zlib) [10671.586517] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [10671.809437] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [10673.511254] Key type lgssc registered [10674.767934] Lustre: Echo OBD driver; http://www.lustre.org/ [10683.395746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10683.418091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.924985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.895330] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10694.121613] LustreError: 358379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.154306] LustreError: 358379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10699.234202] LustreError: 358378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.115836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10700.472552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10705.280801] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10712.883956] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10719.363205] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10725.985514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.355047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10730.482546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10731.862830] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10739.288747] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10755.033903] Lustre: server umount lustre-MDT0000 complete [10756.071901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.089622] LustreError: 360460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.108573] LustreError: 360460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10759.141657] LustreError: 358359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830819 with bad export cookie 2504283838316906378 [10759.145108] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10759.157532] LustreError: 358359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10759.551986] Lustre: server umount lustre-MDT0001 complete [10774.273385] Lustre: server umount lustre-OST0000 complete [10788.542258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10789.118470] LustreError: 362114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10789.235685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10794.214048] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10794.469670] LustreError: 362115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10803.591441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.026963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10809.025800] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10817.844470] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10824.264442] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10832.125701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10832.565405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10834.621365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10839.632537] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10847.849364] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10864.611132] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.624639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.634614] Lustre: Skipped 3 previous similar messages [10864.669284] Lustre: Skipped 2 previous similar messages [10867.233703] Lustre: server umount lustre-MDT0000 complete [10869.730590] LustreError: 362114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10869.761297] LustreError: 362114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10872.025964] LustreError: 362095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830932 with bad export cookie 2504283838316907498 [10872.055404] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10872.368680] Lustre: server umount lustre-MDT0001 complete [10887.948402] Lustre: server umount lustre-OST0000 complete [10904.455132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.061355] LustreError: 366029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10905.217331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10910.360740] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10920.179976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.830292] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10934.747881] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10940.611156] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10948.734716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10949.209904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10949.215249] Lustre: Skipped 1 previous similar message [10950.265570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10955.785511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10964.142376] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10971.109090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10971.111594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.115377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10971.128377] LustreError: Skipped 1 previous similar message [10971.174030] Lustre: Skipped 2 previous similar messages [10971.578843] Lustre: server umount lustre-OST0000 complete [10976.225607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.232516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10976.250272] Lustre: Skipped 1 previous similar message [10976.264892] Lustre: Skipped 2 previous similar messages [10981.344437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10981.355543] Lustre: Skipped 1 previous similar message [10982.282879] Lustre: server umount lustre-MDT0000 complete [10986.469289] LustreError: 366029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10986.483933] LustreError: 366029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10986.495369] LustreError: 366009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831046 with bad export cookie 2504283838316908513 [10986.500197] LustreError: 366009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10986.515281] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10986.797417] Lustre: server umount lustre-MDT0001 complete [11000.434532] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11003.558241] Key type lgssc unregistered [11003.974125] LNet: 369560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11003.987323] LNetError: 369560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11004.013506] LNet: Removed LNI 192.168.204.114@tcp [11005.263737] Key type .llcrypt unregistered [11005.265520] Key type ._llcrypt unregistered [11036.331736] Key type ._llcrypt registered [11036.340406] Key type .llcrypt registered [11036.509529] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11037.771948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11037.799982] alg: No test for adler32 (adler32-zlib) [11038.920763] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [11039.201422] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11040.887758] Key type lgssc registered [11042.227449] Lustre: Echo OBD driver; http://www.lustre.org/ [11050.794070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11050.809766] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.816519] LustreError: 371003:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11050.828935] LustreError: 371003:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11050.836328] LustreError: 371003:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11050.841523] LustreError: 371003:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11050.852054] LustreError: 371003:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11058.377966] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11058.381574] LustreError: 371400:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11058.402637] LustreError: 371400:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11058.407294] LustreError: 371400:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11058.411593] LustreError: 371400:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11058.415896] LustreError: 371400:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11079.636568] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11082.574518] Key type lgssc unregistered [11082.789869] LNet: 372404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11082.798904] LNetError: 372404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11082.814932] LNet: Removed LNI 192.168.204.114@tcp [11083.450757] Key type .llcrypt unregistered [11083.456419] Key type ._llcrypt unregistered [11103.690414] Key type ._llcrypt registered [11103.693371] Key type .llcrypt registered [11103.834556] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [11121.029579] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11122.261680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11122.379133] alg: No test for adler32 (adler32-zlib) [11123.530345] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [11123.725298] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11125.463409] Key type lgssc registered [11126.989173] Lustre: Echo OBD driver; http://www.lustre.org/ [11164.677963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11164.704307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.008581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11166.056625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11166.133320] Lustre: lustre-MDT0000: new disk, initializing [11166.243238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11166.258127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11170.626785] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11180.372521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.483402] Lustre: 375567:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11180.556635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11180.564408] Lustre: Skipped 1 previous similar message [11180.688135] Lustre: lustre-MDT0001: new disk, initializing [11180.853734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11180.928705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11180.948927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11185.240248] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11194.805230] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11201.316886] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11208.993821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.423403] Lustre: lustre-OST0000: new disk, initializing [11209.428733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11209.441740] Lustre: 377545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11209.598906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11215.847274] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11216.432100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11216.450507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11216.595687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11224.424960] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11230.830105] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11231.347442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11236.720050] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11236.875653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11242.482431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11242.497054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11242.507321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11242.507994] Lustre: Skipped 1 previous similar message [11242.552206] Lustre: Skipped 1 previous similar message [11247.585132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11247.593247] Lustre: Skipped 1 previous similar message [11252.705770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11252.709494] Lustre: Skipped 1 previous similar message [11254.239413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11254.531288] Lustre: server umount lustre-OST0000 complete [11262.950152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.958131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11262.963171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11262.964387] Lustre: Skipped 1 previous similar message [11263.003396] Lustre: Skipped 1 previous similar message [11264.396092] Lustre: server umount lustre-MDT0000 complete [11267.950180] LustreError: 376361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831327 with bad export cookie 17476311774033914679 [11267.961755] LustreError: 376361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11267.964955] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11268.363855] Lustre: server umount lustre-MDT0001 complete [11280.038655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11282.040917] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:02:20 (1786831340) [11290.132815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.751541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11296.040614] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11296.228647] LustreError: 380006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.261834] LustreError: 380006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11300.337932] LustreError: 380007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.698683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11305.511750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11310.288759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11318.118875] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11324.654040] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11333.339878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11333.962306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11341.305814] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11349.970038] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11357.242783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11357.501545] Lustre: lustre-OST0001: new disk, initializing [11357.504915] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11357.512276] Lustre: 382843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11357.659347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11364.471770] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11364.556858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11364.568562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11364.662275] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11374.825644] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11380.207456] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11380.216972] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.218633] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11380.240407] Lustre: Skipped 1 previous similar message [11385.158330] Lustre: server umount lustre-OST0001 complete [11385.315860] LustreError: 381828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.333879] LustreError: 381828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11390.439711] LustreError: 381831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.458395] LustreError: 381831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11395.561960] LustreError: 381828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.583370] LustreError: 381828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11401.696681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11401.704887] LustreError: Skipped 1 previous similar message [11401.710772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.735871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11401.744990] Lustre: Skipped 2 previous similar messages [11405.796842] LustreError: 381831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.829243] LustreError: 381831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11407.202709] Lustre: server umount lustre-OST0000 complete [11410.916408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.937856] Lustre: Skipped 1 previous similar message [11417.126943] Lustre: server umount lustre-MDT0000 complete [11420.709172] LustreError: 379988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831480 with bad export cookie 17476311774033915561 [11420.711731] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11420.716815] LustreError: 379988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11427.200673] Lustre: server umount lustre-MDT0001 complete [11438.476284] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11441.416603] Key type lgssc unregistered [11441.827972] LNet: 384832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11441.836422] LNetError: 384832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11441.853188] LNet: Removed LNI 192.168.204.114@tcp [11442.691664] Key type .llcrypt unregistered [11442.693712] Key type ._llcrypt unregistered [11493.197266] Key type ._llcrypt registered [11493.202285] Key type .llcrypt registered [11493.376944] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11494.687749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11494.810804] alg: No test for adler32 (adler32-zlib) [11496.031845] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [11496.332706] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11498.016393] Key type lgssc registered [11499.313051] Lustre: Echo OBD driver; http://www.lustre.org/ [11508.326775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11508.342543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.634631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11509.661742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11509.834522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11514.029963] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11523.186371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11523.267749] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11523.286629] Lustre: 387631:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11523.508386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11528.596797] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11538.051272] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11545.369248] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11549.159830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11549.179824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.211242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11551.201430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.205616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11554.283595] Lustre: server umount lustre-MDT0000 complete [11558.276994] LustreError: 387624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831618 with bad export cookie 9950419695851739559 [11558.284157] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11558.300192] LustreError: 387624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11558.650959] Lustre: server umount lustre-MDT0001 complete [11568.887978] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:07:06 (1786831626) [11576.300824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.779527] LustreError: 390145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11576.795486] LustreError: 390145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11576.837107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11581.643487] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11581.923276] LustreError: 390146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.040150] LustreError: 390145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11590.024841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.303937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11594.659348] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11603.015176] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11608.729618] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11615.650618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11615.876473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11616.127967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11622.846635] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11630.767190] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11640.114358] Lustre: Permanently deactivating lustre-OST0000 [11640.131499] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11650.101165] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11651.527945] LustreError: 393182:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11651.550101] LustreError: 393182:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11651.570539] LustreError: 393182:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11653.945829] LustreError: 393230:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11653.952168] LustreError: 393230:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11662.127849] Lustre: server umount lustre-OST0000 complete [11664.864175] LustreError: 393479:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11664.873870] LustreError: 393479:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11667.425513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11667.444346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.467272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.475635] Lustre: Skipped 1 previous similar message [11671.010253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11671.016721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11671.035240] Lustre: Skipped 1 previous similar message [11676.129519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11676.138112] Lustre: Skipped 1 previous similar message [11681.256176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11681.587945] Lustre: server umount lustre-MDT0000 complete [11685.914321] LustreError: 391992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831745 with bad export cookie 9950419695851740140 [11685.920813] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11685.932552] LustreError: 391992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11695.272685] Lustre: server umount lustre-MDT0001 complete [11704.164234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11709.095816] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11711.753749] LustreError: 394640:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11718.115945] Lustre: 394979:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.114@tcp [11720.169984] Lustre: server umount MGS complete [11731.352127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.999319] LustreError: 395704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11732.034223] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11732.047524] Lustre: Skipped 1 previous similar message [11732.104395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11736.158184] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11737.570386] LustreError: 395704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.585401] LustreError: 395704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11744.834980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11745.241312] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11750.617231] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11759.504714] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11765.835287] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11768.422936] Lustre: Permanently reactivating lustre-OST0000 [11768.438718] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11768.444669] Lustre: Skipped 2 previous similar messages [11799.013649] LustreError: 397535:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11808.768284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11809.032081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.045136] Lustre: Skipped 1 previous similar message [11815.757868] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11824.479963] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11829.731554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11829.740867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11829.743236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11829.766668] Lustre: Skipped 1 previous similar message [11829.794982] Lustre: Skipped 3 previous similar messages [11831.633246] Lustre: server umount lustre-OST0000 complete [11837.407971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11839.987915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.001177] Lustre: Skipped 1 previous similar message [11841.956478] Lustre: server umount lustre-MDT0000 complete [11845.093463] LustreError: 395705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.120492] LustreError: 395705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11846.141309] LustreError: 395686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831906 with bad export cookie 9950419695851741190 [11846.145369] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11846.149263] LustreError: 395686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11846.556751] Lustre: server umount lustre-MDT0001 complete [11857.124482] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11859.940641] Key type lgssc unregistered [11860.151854] LNet: 400122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11860.159300] LNetError: 400122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11860.184609] LNet: Removed LNI 192.168.204.114@tcp [11860.818675] Key type .llcrypt unregistered [11860.820836] Key type ._llcrypt unregistered [11882.362308] Key type ._llcrypt registered [11882.364340] Key type .llcrypt registered [11882.481827] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [11897.970321] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11898.998107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11899.110217] alg: No test for adler32 (adler32-zlib) [11900.292421] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [11900.828473] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11902.503203] Key type lgssc registered [11903.913579] Lustre: Echo OBD driver; http://www.lustre.org/ [11946.754926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11946.799067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.169978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11948.223173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11948.318840] Lustre: lustre-MDT0000: new disk, initializing [11948.428789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11948.475983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11953.846834] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11966.109466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.249905] Lustre: 403278:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11966.300748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11966.305357] Lustre: Skipped 1 previous similar message [11966.363510] Lustre: lustre-MDT0001: new disk, initializing [11966.453275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11966.487800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11966.510442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11971.176961] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11982.366696] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11987.917940] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11994.686164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11994.929784] Lustre: lustre-OST0000: new disk, initializing [11994.932792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11994.937700] Lustre: 405245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11995.011342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12000.933141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12000.954374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12001.058959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12001.438945] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12011.398942] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12016.697867] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12017.019677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12022.361841] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12022.804944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12026.859635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12026.861316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12026.888285] Lustre: Skipped 1 previous similar message [12026.898195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12026.902068] LustreError: Skipped 1 previous similar message [12026.912911] Lustre: Skipped 1 previous similar message [12031.439204] Lustre: server umount lustre-OST0000 complete [12037.088948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12037.111113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12037.120592] Lustre: Skipped 1 previous similar message [12041.950325] Lustre: server umount lustre-MDT0000 complete [12046.512240] LustreError: 403271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832106 with bad export cookie 17426993697016679979 [12046.525366] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12046.526935] LustreError: 403271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12046.882936] Lustre: server umount lustre-MDT0001 complete [12055.328831] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12057.300491] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:15:15 (1786832115) [12065.089744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12065.541933] LustreError: 407708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.553434] LustreError: 407708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12065.610914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12069.735856] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12070.881867] LustreError: 407709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12074.980681] LustreError: 407708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.659323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12078.938783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12083.804563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12092.762520] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12099.125390] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12106.795313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12107.496624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12114.382410] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12122.220814] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12124.248072] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12126.942969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12131.296682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12131.305422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.320978] Lustre: Skipped 1 previous similar message [12131.331380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12134.372980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12134.374163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12134.385211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.403377] Lustre: Skipped 1 previous similar message [12137.242813] Lustre: server umount lustre-OST0000 complete [12144.611257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.627374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12144.634028] Lustre: Skipped 1 previous similar message [12147.958772] Lustre: server umount lustre-MDT0000 complete [12149.728919] LustreError: 407708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.784715] LustreError: 407708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12151.831311] LustreError: 407691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832211 with bad export cookie 17426993697016680861 [12151.840467] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12151.848513] LustreError: 407691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12152.197839] Lustre: server umount lustre-MDT0001 complete [12164.385535] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12167.306369] Key type lgssc unregistered [12167.684638] LNet: 411437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12167.693050] LNetError: 411437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12167.710206] LNet: Removed LNI 192.168.204.114@tcp [12168.431230] Key type .llcrypt unregistered [12168.434662] Key type ._llcrypt unregistered [12200.654904] Key type ._llcrypt registered [12200.657371] Key type .llcrypt registered [12200.805876] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12201.087412] Key type .llcrypt unregistered [12201.089499] Key type ._llcrypt unregistered [12219.589671] Key type ._llcrypt registered [12219.592430] Key type .llcrypt registered [12219.698333] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12221.001552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12221.040810] alg: No test for adler32 (adler32-zlib) [12222.128492] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [12222.421437] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12224.111147] Key type lgssc registered [12225.378275] Lustre: Echo OBD driver; http://www.lustre.org/ [12232.746615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12232.775851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12234.273786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12238.609953] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12244.456288] LustreError: 413650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.479719] LustreError: 413650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12247.064240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12247.454299] LustreError: 413649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.477637] LustreError: 413649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12247.496273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12252.652707] LustreError: 413650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.666404] LustreError: 413650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12254.246897] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12257.764931] LustreError: 413645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.794978] LustreError: 413645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12259.834732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12262.432675] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12264.930061] LustreError: 413649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.956663] LustreError: 413649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12270.395435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.833859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.944447] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12285.409370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12285.422047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12285.454237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12286.436575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12286.438599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12286.453727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.344663] Lustre: server umount lustre-OST0000 complete [12296.673131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.673329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12296.674715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12296.674722] Lustre: Skipped 1 previous similar message [12296.680428] Lustre: Skipped 1 previous similar message [12301.722844] Lustre: server umount lustre-MDT0000 complete [12301.801513] LustreError: 415650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.821459] LustreError: 415650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12305.809716] LustreError: 413629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832365 with bad export cookie 6047409334588983334 [12305.813551] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.824752] LustreError: 413629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12306.185553] Lustre: server umount lustre-MDT0001 complete [12319.717215] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12323.330908] Key type lgssc unregistered [12323.668799] LNet: 416795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12323.677304] LNetError: 416795:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12323.697587] LNet: Removed LNI 192.168.204.114@tcp [12324.520221] Key type .llcrypt unregistered [12324.525994] Key type ._llcrypt unregistered [12349.816539] Key type ._llcrypt registered [12349.822624] Key type .llcrypt registered [12349.962821] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12351.037166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12351.068406] alg: No test for adler32 (adler32-zlib) [12352.242852] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [12352.458789] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12354.119461] Key type lgssc registered [12355.350216] Lustre: Echo OBD driver; http://www.lustre.org/ [12363.847987] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12363.865809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12379.423641] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12384.543374] LustreError: 418188:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [12411.231369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12420.784715] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12430.618934] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12438.211238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12438.809196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12438.892717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12443.345735] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12444.129203] LustreError: 418213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.158063] LustreError: 418213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12445.161093] LustreError: 418214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12445.181832] LustreError: 418214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12450.277930] LustreError: 419229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.292024] LustreError: 419229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12453.119893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12453.530789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12459.048255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12467.761267] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12474.341848] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12483.945122] Lustre: server umount lustre-OST0000 complete [12489.184207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12489.195347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12489.207250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12489.704550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12489.729846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12494.816950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12494.826238] Lustre: Skipped 1 previous similar message [12498.992048] Lustre: server umount lustre-MDT0000 complete [12499.951530] LustreError: 419233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12499.986914] LustreError: 419233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12500.959128] Lustre: 417525:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832544/real 1786832544] req@00000000b72eda7a x1873629583276800/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786832560 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12500.992644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12503.883792] LustreError: 418196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832563 with bad export cookie 9913563566375228364 [12503.892757] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12503.895470] LustreError: 418196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12504.299047] Lustre: server umount lustre-MDT0001 complete [12516.337086] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12520.487700] Key type lgssc unregistered [12520.837046] LNet: 421765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12520.850167] LNetError: 421765:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12520.876629] LNet: Removed LNI 192.168.204.114@tcp [12521.942412] Key type .llcrypt unregistered [12521.944592] Key type ._llcrypt unregistered [12550.617586] Key type ._llcrypt registered [12550.626806] Key type .llcrypt registered [12550.825562] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12552.469098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12552.568890] alg: No test for adler32 (adler32-zlib) [12553.779261] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [12554.161827] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12555.847231] Key type lgssc registered [12557.182649] Lustre: Echo OBD driver; http://www.lustre.org/ [12566.216023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12566.238733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.926755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12573.250845] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12577.254825] LustreError: 423185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12577.271707] LustreError: 423185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12582.373830] LustreError: 423184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.910821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12584.449823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12589.932604] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12599.415799] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12605.050846] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12612.857335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12613.235586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12620.722462] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12630.282374] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12637.828776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12647.390513] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [12648.290966] Lustre: server umount lustre-MDT0000 complete [12648.420498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12648.421694] LustreError: 423180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.432299] Lustre: Skipped 1 previous similar message [12648.454506] LustreError: 423180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12651.185640] LustreError: 423921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.14@tcp arrived at 1786832711 with bad export cookie 1210785010941684011 [12651.195148] LustreError: 423921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12653.539146] LustreError: 423914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12653.558178] LustreError: 423914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12653.656296] LustreError: 423166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832713 with bad export cookie 1210785010941683346 [12653.683347] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12653.692593] LustreError: 423166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12655.585118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12655.589276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.599875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.656862] LustreError: 424967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12658.659194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.679488] LustreError: 424967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12660.022845] Lustre: server umount lustre-OST0000 complete [12667.237517] Lustre: server umount lustre-MDT0001 complete [12677.812359] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12680.800676] Key type lgssc unregistered [12681.111977] LNet: 426892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12681.121331] LNetError: 426892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12681.153214] LNet: Removed LNI 192.168.204.114@tcp [12681.997208] Key type .llcrypt unregistered [12682.002187] Key type ._llcrypt unregistered [12709.663919] Key type ._llcrypt registered [12709.665533] Key type .llcrypt registered [12709.803239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12710.911777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12711.134900] alg: No test for adler32 (adler32-zlib) [12712.158729] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [12712.393098] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12714.087230] Key type lgssc registered [12715.464131] Lustre: Echo OBD driver; http://www.lustre.org/ [12725.334412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12725.364622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12726.881839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12731.884382] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12736.488437] LustreError: 428312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12736.516110] LustreError: 428312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12741.601764] LustreError: 428313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.790554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12744.415898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12750.915263] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12761.132901] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12767.905360] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12775.849662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12776.520225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12786.299151] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12795.355408] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12796.030598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12800.994990] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12801.002644] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12801.010627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12801.513071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12801.516478] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12803.525542] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [12803.531237] Lustre: Skipped 1 previous similar message [12806.627618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12806.645925] Lustre: Skipped 1 previous similar message [12807.096456] Lustre: server umount lustre-MDT0001 complete [12808.597354] LustreError: 430623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.745277] LustreError: 428312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.756331] LustreError: 428312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12816.875315] LustreError: 428313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.876372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12816.937602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12816.965060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12821.383911] Lustre: server umount lustre-OST0000 complete [12827.423971] Lustre: server umount lustre-MDT0000 complete [12843.229753] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12847.017622] Key type lgssc unregistered [12847.334059] LNet: 431999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12847.340176] LNetError: 431999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12847.354419] LNet: Removed LNI 192.168.204.114@tcp [12848.768297] Key type .llcrypt unregistered [12848.775243] Key type ._llcrypt unregistered [12881.857452] Key type ._llcrypt registered [12881.862860] Key type .llcrypt registered [12882.066093] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [12899.548920] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12901.383068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12901.447741] alg: No test for adler32 (adler32-zlib) [12902.638789] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [12902.957902] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12904.698621] Key type lgssc registered [12906.237926] Lustre: Echo OBD driver; http://www.lustre.org/ [12957.042860] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12975.900032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12975.919475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12977.209684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12977.274132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12977.360387] Lustre: lustre-MDT0000: new disk, initializing [12977.531723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12977.554372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12982.798393] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13001.645148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13001.917762] Lustre: lustre-OST0000: new disk, initializing [13001.921688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13001.925723] Lustre: Skipped 1 previous similar message [13001.934322] Lustre: 437038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13002.054743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13009.491359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13009.497224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13009.515957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13010.787571] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13021.855018] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13037.998783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13038.132508] Lustre: 436019:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13038.219187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13038.312446] Lustre: lustre-MDT0001: new disk, initializing [13038.427899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13038.474506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13038.488380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13042.916183] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13053.857458] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13054.214949] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13059.067371] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13059.352600] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13065.649230] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [13069.812177] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13069.813250] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13069.828574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13069.851038] Lustre: Skipped 1 previous similar message [13071.946765] Lustre: server umount lustre-MDT0001 complete [13080.039948] LustreError: 436032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13080.075897] LustreError: 436032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13085.178788] LustreError: 436033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13085.205597] LustreError: 436033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13090.276270] LustreError: 437954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13090.303175] LustreError: 437954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13095.414345] LustreError: 436033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13095.417322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13095.486251] Lustre: Skipped 1 previous similar message [13095.490073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13095.503151] Lustre: Skipped 1 previous similar message [13097.999886] Lustre: server umount lustre-OST0000 complete [13100.512537] LustreError: 438681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13108.237910] Lustre: server umount lustre-MDT0000 complete [13122.404209] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13124.926328] Key type lgssc unregistered [13125.185604] LNet: 440788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13125.190966] LNetError: 440788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13125.209372] LNet: Removed LNI 192.168.204.114@tcp [13126.484254] Key type .llcrypt unregistered [13126.488619] Key type ._llcrypt unregistered [13173.124222] Key type ._llcrypt registered [13173.126481] Key type .llcrypt registered [13173.250869] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13173.660634] Key type .llcrypt unregistered [13173.662826] Key type ._llcrypt unregistered [13191.063169] Key type ._llcrypt registered [13191.065848] Key type .llcrypt registered [13191.184758] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [13192.480862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13192.497864] alg: No test for adler32 (adler32-zlib) [13193.629150] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [13193.861507] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [13195.511766] Key type lgssc registered [13196.870026] Lustre: Echo OBD driver; http://www.lustre.org/ [13224.596862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13224.686924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.931541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13225.970820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13226.066466] Lustre: lustre-MDT0000: new disk, initializing [13226.136463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13226.153535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13230.717069] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13241.397811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13241.528630] Lustre: 444168:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13241.570543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13241.574680] Lustre: Skipped 1 previous similar message [13241.646348] Lustre: lustre-MDT0001: new disk, initializing [13241.735355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13241.788478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13241.797774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13246.004984] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13257.121588] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13264.483284] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13272.220731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13272.580780] Lustre: lustre-OST0000: new disk, initializing [13272.593934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13272.603151] Lustre: 446137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13272.767364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13278.794780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13278.803655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13278.914734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13279.707761] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13290.286566] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13298.146666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13298.159663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13298.179990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.695960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.699317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.714075] Lustre: Skipped 1 previous similar message [13299.723896] Lustre: Skipped 2 previous similar messages [13303.335986] Lustre: server umount lustre-MDT0000 complete [13307.223105] LustreError: 444158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833367 with bad export cookie 4666068112132580072 [13307.233817] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13307.237295] LustreError: 444158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13307.795760] Lustre: server umount lustre-MDT0001 complete [13322.157450] Lustre: server umount lustre-OST0000 complete [13334.899538] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:36:32 (1786833392) [13342.125237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13342.483148] LustreError: 448163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13342.499053] LustreError: 448163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13342.550285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13347.532717] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13347.812144] LustreError: 448164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13352.936288] LustreError: 448163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13355.811531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13356.102723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13361.194663] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13369.586500] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13375.569691] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13395.185260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13395.424376] Lustre: Found index 0 for lustre-OST0000, updating log [13395.623608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.685900] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13409.906800] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13413.045960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13423.586797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13423.594270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13423.603768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13423.613368] Lustre: Skipped 1 previous similar message [13426.974186] Lustre: server umount lustre-OST0000 complete [13443.755724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13443.770270] Lustre: Skipped 1 previous similar message [13443.775488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13443.783785] Lustre: Skipped 2 previous similar messages [13448.160715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13448.170763] LustreError: Skipped 1 previous similar message [13448.173735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13448.182344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13456.031429] LNet: 1 local NIs in recovery (showing 1): 192.168.204.114@tcp [13456.353745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13456.366359] Lustre: Skipped 7 previous similar messages [13458.399406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13458.584508] Lustre: server umount lustre-MDT0000 complete [13461.478056] LustreError: 449404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13461.505402] LustreError: 449404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13462.575987] LustreError: 448145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833522 with bad export cookie 4666068112132581465 [13462.578598] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13462.583957] LustreError: 448145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13463.044508] Lustre: server umount lustre-MDT0001 complete [13473.591727] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:38:51 (1786833531) [14094.156755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14094.716510] LustreError: 454577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14094.793626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14099.098639] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14099.939905] LustreError: 454578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14104.033080] LustreError: 454577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14108.358321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14108.835232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14113.913815] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14121.891989] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14128.344636] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14148.255289] LNet: 1 local NIs in recovery (showing 1): 192.168.204.114@tcp [14148.842505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14149.086960] Lustre: Found index 0 for lustre-OST0000, updating log [14149.590844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14156.245075] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14164.623488] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14166.944469] Lustre: server umount lustre-OST0000 complete [14171.552850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14171.566690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14171.573364] Lustre: Skipped 1 previous similar message [14173.669894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14173.679463] Lustre: Skipped 4 previous similar messages [14176.851833] Lustre: server umount lustre-MDT0000 complete [14177.767703] LustreError: 455579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14177.801852] LustreError: 455579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14180.992866] LustreError: 454558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834241 with bad export cookie 4666068112132582550 [14180.995304] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14181.015529] LustreError: 454558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14181.538404] Lustre: server umount lustre-MDT0001 complete [14208.672151] LNet: 1 local NIs in recovery (showing 1): 192.168.204.114@tcp [14212.039604] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [14222.300940] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14260.512982] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:51:58 (1786834318) [14278.767204] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14282.850383] Key type lgssc unregistered [14283.115621] LNet: 461190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14283.129515] LNetError: 461190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14411.179622] LNet: Removed LNI 192.168.204.114@tcp [14412.471917] Key type .llcrypt unregistered [14412.476812] Key type ._llcrypt unregistered [14427.824911] Key type ._llcrypt registered [14427.826918] Key type .llcrypt registered [14427.927929] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14428.925858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14429.079089] alg: No test for adler32 (adler32-zlib) [14430.240900] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [14430.510919] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [14432.280484] Key type lgssc registered [14433.516074] Lustre: Echo OBD driver; http://www.lustre.org/ [14437.959909] LNet: Added LNI 192.168.204.114@tcp100 [8/256/0/180] [14438.832210] LNet: Added LNI 192.168.204.114@tcp101 [8/256/0/180] [14439.855890] LNet: Added LNI 192.168.204.114@tcp102 [8/256/0/180] [14440.796882] LNet: Added LNI 192.168.204.114@tcp103 [8/256/0/180] [14441.785783] LNet: Added LNI 192.168.204.114@tcp104 [8/256/0/180] [14442.537212] LNet: Added LNI 192.168.204.114@tcp105 [8/256/0/180] [14443.346046] LNet: Added LNI 192.168.204.114@tcp106 [8/256/0/180] [14444.252142] LNet: Added LNI 192.168.204.114@tcp107 [8/256/0/180] [14445.127858] LNet: Added LNI 192.168.204.114@tcp108 [8/256/0/180] [14446.036241] LNet: Added LNI 192.168.204.114@tcp109 [8/256/0/180] [14446.889286] LNet: Added LNI 192.168.204.114@tcp110 [8/256/0/180] [14447.777756] LNet: Added LNI 192.168.204.114@tcp111 [8/256/0/180] [14448.822602] LNet: Added LNI 192.168.204.114@tcp112 [8/256/0/180] [14449.823738] LNet: Added LNI 192.168.204.114@tcp113 [8/256/0/180] [14450.753800] LNet: Added LNI 192.168.204.114@tcp114 [8/256/0/180] [14451.611000] LNet: Added LNI 192.168.204.114@tcp115 [8/256/0/180] [14452.467814] LNet: Added LNI 192.168.204.114@tcp116 [8/256/0/180] [14453.406775] LNet: Added LNI 192.168.204.114@tcp117 [8/256/0/180] [14454.302842] LNet: Added LNI 192.168.204.114@tcp118 [8/256/0/180] [14455.192651] LNet: Added LNI 192.168.204.114@tcp119 [8/256/0/180] [14456.253862] LNet: Added LNI 192.168.204.114@tcp120 [8/256/0/180] [14457.117500] LNet: Added LNI 192.168.204.114@tcp121 [8/256/0/180] [14457.942871] LNet: Added LNI 192.168.204.114@tcp122 [8/256/0/180] [14458.793923] LNet: Added LNI 192.168.204.114@tcp123 [8/256/0/180] [14459.737657] LNet: Added LNI 192.168.204.114@tcp124 [8/256/0/180] [14460.698906] LNet: Added LNI 192.168.204.114@tcp125 [8/256/0/180] [14461.553718] LNet: Added LNI 192.168.204.114@tcp126 [8/256/0/180] [14462.558520] LNet: Added LNI 192.168.204.114@tcp127 [8/256/0/180] [14463.499646] LNet: Added LNI 192.168.204.114@tcp128 [8/256/0/180] [14464.382649] LNet: Added LNI 192.168.204.114@tcp129 [8/256/0/180] [14465.388783] LNet: Added LNI 192.168.204.114@tcp130 [8/256/0/180] [14466.437357] LNet: Added LNI 192.168.204.114@tcp131 [8/256/0/180] [14467.560441] LNet: Added LNI 192.168.204.114@tcp132 [8/256/0/180] [14468.495847] LNet: Added LNI 192.168.204.114@tcp133 [8/256/0/180] [14469.418536] LNet: Added LNI 192.168.204.114@tcp134 [8/256/0/180] [14470.282445] LNet: Added LNI 192.168.204.114@tcp135 [8/256/0/180] [14477.180652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14477.251311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14478.393592] Lustre: 464223:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.114@tcp: can't add NID 192.168.204.114@tcp130: rc = -75 [14478.522153] LNetError: 464253:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14478.543316] LNetError: Protocol error connecting to 192.168.204.114@tcp119 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14478.684668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14478.687119] LNet: 6 local NIs in recovery (showing 6): 192.168.204.114@tcp119, 192.168.204.114@tcp127, 192.168.204.114@tcp128, 192.168.204.114@tcp129, 192.168.204.114@tcp126, 192.168.204.114@tcp121 [14478.723398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14478.800672] Lustre: lustre-MDT0000: new disk, initializing [14478.902265] Lustre: 464280:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.114@tcp: can't add NID 192.168.204.114@tcp130: rc = -75 [14478.911777] Lustre: 464280:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14478.947047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14478.982412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14479.777938] LNetError: 461681:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14479.803768] LNetError: 461681:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [14479.815489] LNetError: Protocol error connecting to 192.168.204.114@tcp119 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14479.827605] LNetError: Skipped 10 previous similar messages [14481.836975] LNetError: 464252:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14481.852456] LNetError: Protocol error connecting to 192.168.204.114@tcp119 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14483.536760] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14485.926583] LNetError: 464253:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14485.942996] LNetError: Protocol error connecting to 192.168.204.114@tcp119 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14493.089917] LNetError: 464243:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14493.109872] LNetError: Protocol error connecting to 192.168.204.114@tcp119 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14493.837243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14494.064415] Lustre: 464241:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14494.126228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14494.130889] Lustre: Skipped 1 previous similar message [14494.211252] Lustre: lustre-MDT0001: new disk, initializing [14494.252283] Lustre: 465045:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.114@tcp: can't add NID 192.168.204.114@tcp130: rc = -75 [14494.258747] Lustre: 465045:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14494.395298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14494.435505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14494.452552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14499.225332] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14500.928035] Lustre: 465386:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.114@tcp: can't add NID 192.168.204.114@tcp130: rc = -75 [14500.941842] Lustre: 465386:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14509.349114] LNetError: 461682:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14509.365465] LNetError: Protocol error connecting to 192.168.204.114@tcp125 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14509.896688] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14515.820673] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14523.657508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14524.111678] Lustre: lustre-OST0000: new disk, initializing [14524.121496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14524.139862] Lustre: 466217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14524.261865] Lustre: 466222:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.114@tcp: can't add NID 192.168.204.114@tcp130: rc = -75 [14524.274494] Lustre: 466222:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14524.318271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14525.732330] LNetError: 464252:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14525.759905] LNetError: 464252:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14525.775349] LNetError: Protocol error connecting to 192.168.204.114@tcp123 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14525.791960] LNetError: Skipped 7 previous similar messages [14530.765319] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14533.156194] Lustre: 466674:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.114@tcp: can't add NID 192.168.204.114@tcp130: rc = -75 [14533.168944] Lustre: 466674:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14533.220580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14533.228853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14533.345336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14538.144207] LNet: 1 local NIs in recovery (showing 1): 192.168.204.114@tcp119 [14539.684830] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14548.968850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14548.969725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14548.975595] LustreError: Skipped 1 previous similar message [14548.985615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14548.992068] Lustre: Skipped 1 previous similar message [14553.712829] Lustre: server umount lustre-OST0000 complete [14559.201978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14559.209717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14559.219430] Lustre: Skipped 2 previous similar messages [14559.241739] Lustre: Skipped 1 previous similar message [14559.525879] LNetError: 464243:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.114 [14559.532881] LNetError: 464243:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [14559.538274] LNetError: Protocol error connecting to 192.168.204.114@tcp122 at host 192.168.204.114:988: is it running a compatible version of Lustre? [14559.554175] LNetError: Skipped 12 previous similar messages [14563.912434] Lustre: server umount lustre-MDT0000 complete [14567.483929] LustreError: 465044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834627 with bad export cookie 10293973229083593715 [14567.486941] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14567.506436] LustreError: 465044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14567.973232] Lustre: server umount lustre-MDT0001 complete [14579.902551] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14582.829463] Key type lgssc unregistered [14583.271540] LNet: 468132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14583.287258] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14583.296459] LNet: Removed LNI 192.168.204.114@tcp [14584.351729] LNet: Removed LNI 192.168.204.114@tcp100 [14584.356425] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14584.369520] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14584.383339] LNet: Removed LNI 192.168.204.114@tcp101 [14585.439182] LNet: Removed LNI 192.168.204.114@tcp102 [14585.441155] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14585.445464] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14585.449793] LNet: Removed LNI 192.168.204.114@tcp103 [14586.464049] LNet: Removed LNI 192.168.204.114@tcp104 [14587.487683] LNet: Removed LNI 192.168.204.114@tcp105 [14587.497443] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14587.513455] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14587.524100] LNet: Removed LNI 192.168.204.114@tcp106 [14588.575204] LNet: Removed LNI 192.168.204.114@tcp107 [14589.599155] LNet: Removed LNI 192.168.204.114@tcp108 [14590.623162] LNet: Removed LNI 192.168.204.114@tcp109 [14591.647925] LNet: Removed LNI 192.168.204.114@tcp110 [14591.652932] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14591.662198] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14591.671240] LNet: Removed LNI 192.168.204.114@tcp111 [14592.735590] LNet: Removed LNI 192.168.204.114@tcp112 [14593.759154] LNet: Removed LNI 192.168.204.114@tcp113 [14594.783336] LNet: Removed LNI 192.168.204.114@tcp114 [14595.807188] LNet: Removed LNI 192.168.204.114@tcp115 [14596.832307] LNet: Removed LNI 192.168.204.114@tcp116 [14597.856252] LNet: Removed LNI 192.168.204.114@tcp117 [14598.879142] LNet: Removed LNI 192.168.204.114@tcp118 [14598.887994] LNet: Removed LNI 192.168.204.114@tcp119 [14599.969532] LNet: Removed LNI 192.168.204.114@tcp120 [14599.983302] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14600.000365] LNetError: 468132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [14600.007268] LNet: Removed LNI 192.168.204.114@tcp121 [14601.056646] LNet: Removed LNI 192.168.204.114@tcp122 [14602.080360] LNet: Removed LNI 192.168.204.114@tcp123 [14603.103159] LNet: Removed LNI 192.168.204.114@tcp124 [14604.128224] LNet: Removed LNI 192.168.204.114@tcp125 [14605.151888] LNet: Removed LNI 192.168.204.114@tcp126 [14606.175184] LNet: Removed LNI 192.168.204.114@tcp127 [14607.199160] LNet: Removed LNI 192.168.204.114@tcp128 [14608.223377] LNet: Removed LNI 192.168.204.114@tcp129 [14608.229190] LNet: Removed LNI 192.168.204.114@tcp130 [14608.235452] LNet: Removed LNI 192.168.204.114@tcp131 [14608.245812] LNet: Removed LNI 192.168.204.114@tcp132 [14608.250726] LNet: Removed LNI 192.168.204.114@tcp133 [14608.258447] LNet: Removed LNI 192.168.204.114@tcp134 [14608.272566] LNet: Removed LNI 192.168.204.114@tcp135 [14608.928631] Key type .llcrypt unregistered [14608.930329] Key type ._llcrypt unregistered [14630.402687] Key type ._llcrypt registered [14630.407119] Key type .llcrypt registered [14630.523237] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [14643.936389] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14644.743365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14644.800149] alg: No test for adler32 (adler32-zlib) [14645.814624] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [14645.996943] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [14647.687239] Key type lgssc registered [14648.524851] Lustre: Echo OBD driver; http://www.lustre.org/ [14678.811708] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:58:57 (1786834737) [14695.957863] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [14698.712335] Key type lgssc unregistered [14699.049095] LNet: 472217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14699.053070] LNetError: 472217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14699.072511] LNet: Removed LNI 192.168.204.114@tcp [14699.865616] Key type .llcrypt unregistered [14699.868208] Key type ._llcrypt unregistered [14715.467455] Key type ._llcrypt registered [14715.473927] Key type .llcrypt registered [14715.625669] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [14716.490455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14716.654424] alg: No test for adler32 (adler32-zlib) [14717.677674] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [14717.920697] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [14719.610086] Key type lgssc registered [14720.627298] Lustre: Echo OBD driver; http://www.lustre.org/ [14729.077925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14729.103496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14730.321116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14730.342781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14730.395464] Lustre: lustre-MDT0000: new disk, initializing [14730.456678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14730.482430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14734.683862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14744.592908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14744.732951] Lustre: 473428:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14744.795957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14744.798748] Lustre: Skipped 1 previous similar message [14744.890377] Lustre: lustre-MDT0001: new disk, initializing [14744.990114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14745.051167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14745.066931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14749.025491] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14757.208822] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14762.026273] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14768.018155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14768.315959] Lustre: lustre-OST0000: new disk, initializing [14768.319950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14768.328628] Lustre: 475398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14768.444433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14774.246832] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [14774.860615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14774.868514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14774.969036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14782.358124] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14785.868256] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing lnet_if_list [14790.417519] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14790.442828] LNet: Added LNI 192.168.204.114@tcp10 [8/256/0/180] [14795.409717] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14795.457662] LNet: Added LNI 192.168.204.114@tcp11 [8/256/0/180] [14799.086620] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14799.120831] LNet: Added LNI 192.168.204.114@tcp12 [8/256/0/180] [14803.657810] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14803.680135] LNet: Added LNI 192.168.204.114@tcp13 [8/256/0/180] [14807.212213] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14807.239468] LNet: Added LNI 192.168.204.114@tcp14 [8/256/0/180] [14811.169578] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14811.189723] LNet: Added LNI 192.168.204.114@tcp15 [8/256/0/180] [14815.122847] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14815.150542] LNet: Added LNI 192.168.204.114@tcp16 [8/256/0/180] [14819.178745] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14819.206824] LNet: Added LNI 192.168.204.114@tcp17 [8/256/0/180] [14823.951357] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14823.978811] LNet: Added LNI 192.168.204.114@tcp18 [8/256/0/180] [14828.338767] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14828.379816] LNet: Added LNI 192.168.204.114@tcp19 [8/256/0/180] [14832.984704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14833.009653] LNet: Added LNI 192.168.204.114@tcp20 [8/256/0/180] [14837.637541] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14837.659145] LNet: Added LNI 192.168.204.114@tcp21 [8/256/0/180] [14842.031805] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14842.051377] LNet: Added LNI 192.168.204.114@tcp22 [8/256/0/180] [14846.687753] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14846.715712] LNet: Added LNI 192.168.204.114@tcp23 [8/256/0/180] [14851.813780] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14851.834695] LNet: Added LNI 192.168.204.114@tcp24 [8/256/0/180] [14856.766191] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14856.803938] LNet: Added LNI 192.168.204.114@tcp25 [8/256/0/180] [14861.676692] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14861.699970] LNet: Added LNI 192.168.204.114@tcp26 [8/256/0/180] [14866.829249] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14866.857274] LNet: Added LNI 192.168.204.114@tcp27 [8/256/0/180] [14872.106924] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14872.134823] LNet: Added LNI 192.168.204.114@tcp28 [8/256/0/180] [14877.181391] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14877.205925] LNet: Added LNI 192.168.204.114@tcp29 [8/256/0/180] [14882.652514] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14882.690369] LNet: Added LNI 192.168.204.114@tcp30 [8/256/0/180] [14888.065703] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14888.091523] LNet: Added LNI 192.168.204.114@tcp31 [8/256/0/180] [14893.631781] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14893.652107] LNet: Added LNI 192.168.204.114@tcp32 [8/256/0/180] [14899.968038] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14899.993175] LNet: Added LNI 192.168.204.114@tcp33 [8/256/0/180] [14906.123707] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14906.157985] LNet: Added LNI 192.168.204.114@tcp34 [8/256/0/180] [14912.464816] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14912.492448] LNet: Added LNI 192.168.204.114@tcp35 [8/256/0/180] [14919.759373] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14919.801433] LNet: Added LNI 192.168.204.114@tcp36 [8/256/0/180] [14926.236974] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14926.270052] LNet: Added LNI 192.168.204.114@tcp37 [8/256/0/180] [14933.166584] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14933.212675] LNet: Added LNI 192.168.204.114@tcp38 [8/256/0/180] [14939.665335] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14939.701850] LNet: Added LNI 192.168.204.114@tcp39 [8/256/0/180] [14946.400431] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14946.429131] LNet: Added LNI 192.168.204.114@tcp40 [8/256/0/180] [14953.013630] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14953.061342] LNet: Added LNI 192.168.204.114@tcp41 [8/256/0/180] [14959.646468] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14959.669583] LNet: Added LNI 192.168.204.114@tcp42 [8/256/0/180] [14966.606140] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14966.653996] LNet: Added LNI 192.168.204.114@tcp43 [8/256/0/180] [14973.799556] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14973.840708] LNet: Added LNI 192.168.204.114@tcp44 [8/256/0/180] [14981.321666] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14981.349950] LNet: Added LNI 192.168.204.114@tcp45 [8/256/0/180] [14987.853349] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14987.896329] LNet: Added LNI 192.168.204.114@tcp46 [8/256/0/180] [14994.923585] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14994.979844] LNet: Added LNI 192.168.204.114@tcp47 [8/256/0/180] [15002.538483] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15002.577386] LNet: Added LNI 192.168.204.114@tcp48 [8/256/0/180] [15010.030760] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15010.061309] LNet: Added LNI 192.168.204.114@tcp49 [8/256/0/180] [15017.912416] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15017.940660] LNet: Added LNI 192.168.204.114@tcp50 [8/256/0/180] [15036.384506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15036.388647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15036.420455] Lustre: Skipped 2 previous similar messages [15036.432140] Lustre: Skipped 2 previous similar messages [15040.800366] Lustre: server umount lustre-MDT0000 complete [15044.203942] LustreError: 475397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835104 with bad export cookie 14139675705947942557 [15044.216021] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15044.220727] LustreError: 475397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15044.643377] Lustre: server umount lustre-MDT0001 complete [15059.604917] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [15062.945121] Lustre: 472703:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835106/real 1786835106] req@0000000053acf5f3 x1873632063277056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786835122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15062.971311] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15065.055353] Lustre: 472704:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835109/real 1786835109] req@00000000709fcc7e x1873632063277184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786835125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15066.703818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15067.103186] Lustre: 472705:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835111/real 1786835111] req@000000009fa22aee x1873632063277312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786835127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15069.159313] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc43a3823ac3f788a [15069.173415] Lustre: MGC192.168.204.114@tcp: Connection restored to 0@lo (at 0@lo) [15069.419944] LustreError: 475391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15069.441930] LustreError: 475391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [15069.483783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15070.111141] Lustre: 472703:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835114/real 1786835114] req@00000000d2012b03 x1873632063277568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786835130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15071.529329] LustreError: 475392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15073.453511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15073.570133] LustreError: 499619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15073.591518] LustreError: 499619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15075.807165] Lustre: 472702:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835119/real 1786835119] req@00000000a6320d75 x1873632063277696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786835135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15078.886207] LustreError: 499903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15078.928285] LustreError: 499903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15080.663881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15080.994514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15084.857192] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15086.065846] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15091.184255] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15091.967264] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15097.348875] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15115.755061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15115.760099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15115.769962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15116.770659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15116.775036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15116.783215] Lustre: Skipped 1 previous similar message [15121.903089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15121.907783] Lustre: Skipped 1 previous similar message [15127.014706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15127.026146] Lustre: Skipped 1 previous similar message [15128.031138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15128.213439] Lustre: server umount lustre-OST0000 complete [15132.130697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15132.132130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15132.154768] Lustre: Skipped 1 previous similar message [15137.248886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15137.255348] Lustre: Skipped 3 previous similar messages [15137.728249] Lustre: server umount lustre-MDT0000 complete [15141.116982] LustreError: 473420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835201 with bad export cookie 14139675705947945098 [15141.118616] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15141.128245] LustreError: 473420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15141.407667] Lustre: server umount lustre-MDT0001 complete [15151.487080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [15153.884744] Key type lgssc unregistered [15154.241777] LNet: 502402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15154.247906] LNetError: 502402:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15154.252269] LNet: Removed LNI 192.168.204.114@tcp [15154.255259] LNet: Removed LNI 192.168.204.114@tcp10 [15154.258654] LNet: Removed LNI 192.168.204.114@tcp11 [15154.260750] LNet: Removed LNI 192.168.204.114@tcp12 [15154.264273] LNet: Removed LNI 192.168.204.114@tcp13 [15154.266924] LNet: Removed LNI 192.168.204.114@tcp14 [15154.269704] LNet: Removed LNI 192.168.204.114@tcp15 [15154.272494] LNet: Removed LNI 192.168.204.114@tcp16 [15154.274742] LNet: Removed LNI 192.168.204.114@tcp17 [15154.277498] LNet: Removed LNI 192.168.204.114@tcp18 [15154.279797] LNet: Removed LNI 192.168.204.114@tcp19 [15154.282241] LNet: Removed LNI 192.168.204.114@tcp20 [15154.284517] LNet: Removed LNI 192.168.204.114@tcp21 [15154.288297] LNet: Removed LNI 192.168.204.114@tcp22 [15154.290535] LNet: Removed LNI 192.168.204.114@tcp23 [15154.292964] LNet: Removed LNI 192.168.204.114@tcp24 [15154.294681] LNet: Removed LNI 192.168.204.114@tcp25 [15154.296991] LNet: Removed LNI 192.168.204.114@tcp26 [15154.299292] LNet: Removed LNI 192.168.204.114@tcp27 [15154.301517] LNet: Removed LNI 192.168.204.114@tcp28 [15154.303907] LNet: Removed LNI 192.168.204.114@tcp29 [15154.306776] LNet: Removed LNI 192.168.204.114@tcp30 [15154.309124] LNet: Removed LNI 192.168.204.114@tcp31 [15154.311521] LNet: Removed LNI 192.168.204.114@tcp32 [15154.313917] LNet: Removed LNI 192.168.204.114@tcp33 [15154.316228] LNet: Removed LNI 192.168.204.114@tcp34 [15154.318677] LNet: Removed LNI 192.168.204.114@tcp35 [15154.321655] LNet: Removed LNI 192.168.204.114@tcp36 [15154.325471] LNet: Removed LNI 192.168.204.114@tcp37 [15154.327908] LNet: Removed LNI 192.168.204.114@tcp38 [15154.330748] LNet: Removed LNI 192.168.204.114@tcp39 [15154.333522] LNet: Removed LNI 192.168.204.114@tcp40 [15154.335782] LNet: Removed LNI 192.168.204.114@tcp41 [15154.339187] LNet: Removed LNI 192.168.204.114@tcp42 [15154.341239] LNet: Removed LNI 192.168.204.114@tcp43 [15154.343495] LNet: Removed LNI 192.168.204.114@tcp44 [15154.346199] LNet: Removed LNI 192.168.204.114@tcp45 [15154.347978] LNet: Removed LNI 192.168.204.114@tcp46 [15154.350221] LNet: Removed LNI 192.168.204.114@tcp47 [15154.352766] LNet: Removed LNI 192.168.204.114@tcp48 [15154.355450] LNet: Removed LNI 192.168.204.114@tcp49 [15154.364198] LNet: Removed LNI 192.168.204.114@tcp50 [15155.002893] Key type .llcrypt unregistered [15155.005631] Key type ._llcrypt unregistered [15172.850274] Key type ._llcrypt registered [15172.852455] Key type .llcrypt registered [15172.956875] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [15185.341348] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [15186.105727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15186.200456] alg: No test for adler32 (adler32-zlib) [15187.219681] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [15187.455044] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [15189.119243] Key type lgssc registered [15189.976895] Lustre: Echo OBD driver; http://www.lustre.org/ [15220.625722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15220.651281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15221.883916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15221.929713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15222.057279] Lustre: lustre-MDT0000: new disk, initializing [15222.129647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15222.155681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15225.855833] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15234.121653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15234.249626] Lustre: 505578:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15234.288692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15234.291656] Lustre: Skipped 1 previous similar message [15234.342381] Lustre: lustre-MDT0001: new disk, initializing [15234.414275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15234.441432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15234.456250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15238.107626] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15245.712326] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15250.570663] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15256.537275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15256.752977] Lustre: *** cfs_fail_loc=90f, val=0*** [15256.840164] Lustre: lustre-OST0000: new disk, initializing [15256.844637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15256.853391] Lustre: 507593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15256.969507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15262.471995] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15263.306038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15263.315222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15263.460530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15270.136378] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15279.075989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15279.076309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15279.077477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15279.081385] LustreError: Skipped 1 previous similar message [15279.103070] Lustre: Skipped 1 previous similar message [15284.199973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15284.207547] Lustre: Skipped 2 previous similar messages [15289.316273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15289.323552] Lustre: Skipped 1 previous similar message [15290.335146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15290.427160] Lustre: server umount lustre-OST0000 complete [15294.435408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15294.438911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15294.452338] Lustre: Skipped 1 previous similar message [15299.557343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15299.565336] Lustre: Skipped 2 previous similar messages [15299.716217] Lustre: server umount lustre-MDT0000 complete [15302.762290] LustreError: 505572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835362 with bad export cookie 4941282934071072655 [15302.762698] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15302.769133] LustreError: 505572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15303.037820] Lustre: server umount lustre-MDT0001 complete [15311.640287] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [15313.958698] Key type lgssc unregistered [15314.229648] LNet: 509498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15314.240876] LNetError: 509498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15314.263607] LNet: Removed LNI 192.168.204.114@tcp [15315.016319] Key type .llcrypt unregistered [15315.021138] Key type ._llcrypt unregistered [15329.692485] Key type ._llcrypt registered [15329.693914] Key type .llcrypt registered [15329.766840] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [15330.299464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15330.384221] alg: No test for adler32 (adler32-zlib) [15331.367618] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [15331.558642] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [15333.216158] Key type lgssc registered [15334.086818] Lustre: Echo OBD driver; http://www.lustre.org/ [15340.009612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15340.020201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15341.316534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15344.366378] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15350.533095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15350.814949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15354.260679] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15359.956194] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15364.235682] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15368.741679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15368.987368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15373.486750] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15379.191918] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15396.703889] loop0: detected capacity change from 0 to 204800000 [15396.852233] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15396.863995] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15396.871206] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15401.359248] loop0: detected capacity change from 0 to 204800000 [15401.558274] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15401.574208] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15401.581814] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15405.640331] loop0: detected capacity change from 0 to 204800000 [15409.639427] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15409.778923] Lustre: fs15246-OST0000: new disk, initializing [15409.783065] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15409.789200] Lustre: 514364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15409.837370] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15414.785634] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15422.882697] loop1: detected capacity change from 0 to 204800000 [15426.828202] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15426.901756] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15426.948469] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15427.013175] Lustre: fs15246-MDT0000: new disk, initializing [15427.114650] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15427.142390] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15427.206720] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15428.257095] Lustre: 515485:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000fc592638 x1873632706997120/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786835504 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15430.107738] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15437.349669] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15437.354839] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15437.378945] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15445.487089] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15445.503218] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15450.600559] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15451.511478] Lustre: server umount fs15246-MDT0000 complete [15455.721917] LustreError: 510741:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15455.737392] LustreError: 510741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15458.485747] Lustre: server umount fs15246-OST0000 complete [15467.489023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15467.498302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15467.521662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15468.515463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15468.524968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15471.079122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15471.085303] Lustre: Skipped 2 previous similar messages [15473.312615] Lustre: server umount lustre-OST0000 complete [15476.196649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15476.200637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15476.202499] Lustre: Skipped 1 previous similar message [15476.208674] Lustre: Skipped 1 previous similar message [15482.581432] Lustre: server umount lustre-MDT0000 complete [15485.605115] LustreError: 510726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835545 with bad export cookie 7081369091514615154 [15485.614419] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15485.617497] LustreError: 510726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15485.875261] Lustre: server umount lustre-MDT0001 complete [15494.596549] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [15496.735729] Key type lgssc unregistered [15497.003717] LNet: 518578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15497.016307] LNetError: 518578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15497.030890] LNet: Removed LNI 192.168.204.114@tcp [15497.682623] Key type .llcrypt unregistered [15497.684491] Key type ._llcrypt unregistered [15511.057219] Key type ._llcrypt registered [15511.060581] Key type .llcrypt registered [15511.169977] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [15512.044344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15512.065222] alg: No test for adler32 (adler32-zlib) [15513.085274] Lustre: Lustre: Build Version: 2.17.54_84_gc7429ec [15513.274242] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [15514.919179] Key type lgssc registered [15515.733174] Lustre: Echo OBD driver; http://www.lustre.org/ [15545.827143] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [15553.318989] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [15580.172507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15580.205289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15581.373633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15581.395803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15581.457491] Lustre: lustre-MDT0000: new disk, initializing [15581.516533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15581.544210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15584.703767] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15592.629503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15592.760966] Lustre: 523019:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15592.833532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15592.842033] Lustre: Skipped 1 previous similar message [15592.943560] Lustre: lustre-MDT0001: new disk, initializing [15593.030942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15593.088744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15593.111293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15596.469392] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15603.742715] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15607.739192] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15612.317171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15612.538172] Lustre: lustre-OST0000: new disk, initializing [15612.545301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15612.552268] Lustre: 524987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15612.630905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15616.765912] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [15622.181055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15622.186141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15622.269937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15623.190316] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15626.078586] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15626.226307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15629.163560] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15629.316671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15632.355933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15632.361173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15632.362989] LustreError: Skipped 1 previous similar message [15632.384583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15632.395189] Lustre: Skipped 1 previous similar message [15636.690458] Lustre: server umount lustre-OST0000 complete [15642.593433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15642.595114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15642.600971] Lustre: Skipped 2 previous similar messages [15647.713147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15647.722124] Lustre: Skipped 2 previous similar messages [15652.833485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15652.839701] Lustre: Skipped 1 previous similar message [15653.343122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15653.461873] Lustre: server umount lustre-MDT0000 complete [15655.832707] LustreError: 523012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835715 with bad export cookie 7502709281103485994 [15655.840898] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15655.846734] LustreError: 523012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15656.050936] Lustre: server umount lustre-MDT0001 complete [15671.814169] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [15679.033820] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [15700.147534] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15424 sec ========= 19:15:58 (1786835758) [15701.440820] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:16:00 (1786835760) === [15703.833193] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:16:02 (1786835762) === [15728.375361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [15730.374824] Key type lgssc unregistered [15730.606807] LNet: 531628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15730.613643] LNetError: 531628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15730.626509] LNet: Removed LNI 192.168.204.114@tcp [15731.230763] Key type .llcrypt unregistered [15731.233149] Key type ._llcrypt unregistered