[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507603700 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013078] *** VALIDATE selinux *** [ 0.021830] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025718] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] *** VALIDATE tmpfs *** [ 0.030459] *** VALIDATE proc *** [ 0.031263] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039836] debug: unmapping init [mem 0xffffffff89c59000-0xffffffff89c60fff] [ 0.041916] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049015] ... event mask: 000000070000000f [ 0.051208] rcu: Hierarchical SRCU implementation. [ 0.053456] smp: Bringing up secondary CPUs ... [ 0.054620] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.059015] smp: Brought up 1 node, 4 CPUs [ 0.061022] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130276] node 0 deferred pages initialised in 66ms [ 0.134167] devtmpfs: initialized [ 0.135243] x86/mm: Memory block size: 128MB [ 0.138627] gcov: version magic: 0x41383552 [ 0.140476] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142353] pinctrl core: initialized pinctrl subsystem [ 0.143256] [ 0.143880] ************************************************************* [ 0.144019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145016] ** ** [ 0.146017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147022] ** ** [ 0.148033] ** This means that this kernel is built to expose internal ** [ 0.149017] ** IOMMU data structures, which may compromise security on ** [ 0.150018] ** your system. ** [ 0.151016] ** ** [ 0.152020] ** If you see this message and you are not debugging the ** [ 0.153016] ** kernel, report this immediately to your vendor! ** [ 0.154021] ** ** [ 0.155016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156014] ************************************************************* [ 0.157841] NET: Registered protocol family 16 [ 0.158608] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161597] cpuidle: using governor menu [ 0.163449] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166883] PCI: Using configuration type 1 for base access [ 0.169152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182124] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187131] cryptd: max_cpu_qlen set to 1000 [ 0.190004] ACPI: Added _OSI(Module Device) [ 0.191018] ACPI: Added _OSI(Processor Device) [ 0.192012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204844] ACPI: Interpreter enabled [ 0.206077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207013] ACPI: Using IOAPIC for interrupt routing [ 0.209150] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212449] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223961] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236674] acpiphp: Slot [2] registered [ 0.238168] acpiphp: Slot [5] registered [ 0.240144] acpiphp: Slot [6] registered [ 0.241176] acpiphp: Slot [7] registered [ 0.243141] acpiphp: Slot [8] registered [ 0.244149] acpiphp: Slot [9] registered [ 0.246201] acpiphp: Slot [10] registered [ 0.247143] acpiphp: Slot [3] registered [ 0.249175] acpiphp: Slot [4] registered [ 0.250164] acpiphp: Slot [11] registered [ 0.252218] acpiphp: Slot [12] registered [ 0.253126] acpiphp: Slot [13] registered [ 0.254114] acpiphp: Slot [14] registered [ 0.256134] acpiphp: Slot [15] registered [ 0.258109] acpiphp: Slot [16] registered [ 0.259142] acpiphp: Slot [17] registered [ 0.261115] acpiphp: Slot [18] registered [ 0.262111] acpiphp: Slot [19] registered [ 0.263125] acpiphp: Slot [20] registered [ 0.265109] acpiphp: Slot [21] registered [ 0.267114] acpiphp: Slot [22] registered [ 0.268108] acpiphp: Slot [23] registered [ 0.269124] acpiphp: Slot [24] registered [ 0.271103] acpiphp: Slot [25] registered [ 0.272096] acpiphp: Slot [26] registered [ 0.274135] acpiphp: Slot [27] registered [ 0.275107] acpiphp: Slot [28] registered [ 0.276084] acpiphp: Slot [29] registered [ 0.277115] acpiphp: Slot [30] registered [ 0.278146] acpiphp: Slot [31] registered [ 0.279067] PCI host bridge to bus 0000:00 [ 0.281021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.291036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.294037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.295186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298382] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302474] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.311025] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.316522] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327699] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329876] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335979] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.353018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.360021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366508] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.383019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424719] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.432020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.464020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.471000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.476021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.482020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509756] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.524019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.549658] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.584026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.596585] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.614020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.643016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.662604] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.665380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.668468] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.671395] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.674249] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.678199] iommu: Default domain type: Passthrough [ 0.679551] SCSI subsystem initialized [ 0.681064] ACPI: bus type USB registered [ 0.682152] usbcore: registered new interface driver usbfs [ 0.684115] usbcore: registered new interface driver hub [ 0.687121] usbcore: registered new device driver usb [ 0.689209] pps_core: LinuxPPS API ver. 1 registered [ 0.690017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.693086] PTP clock support registered [ 0.695152] EDAC MC: Ver: 3.0.0 [ 0.698118] PCI: Using ACPI for IRQ routing [ 0.699945] NetLabel: Initializing [ 0.702012] NetLabel: domain hash size = 128 [ 0.703011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.706146] NetLabel: unlabeled traffic allowed by default [ 0.708124] vgaarb: loaded [ 0.710255] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.712015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.720000] clocksource: Switched to clocksource kvm-clock [ 0.831368] VFS: Disk quotas dquot_6.6.0 [ 0.833254] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.836224] *** VALIDATE ramfs *** [ 0.837620] *** VALIDATE hugetlbfs *** [ 0.840130] pnp: PnP ACPI init [ 0.842867] pnp: PnP ACPI: found 6 devices [ 0.880820] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.884433] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.886922] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.889407] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.892401] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895375] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.898729] NET: Registered protocol family 2 [ 0.901633] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.907098] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.910380] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.915396] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.918754] TCP: Hash tables configured (established 65536 bind 65536) [ 0.922439] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.926735] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.930019] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933469] NET: Registered protocol family 1 [ 0.936403] RPC: Registered named UNIX socket transport module. [ 0.938824] RPC: Registered udp transport module. [ 0.940432] RPC: Registered tcp transport module. [ 0.942553] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.945226] NET: Registered protocol family 44 [ 0.947300] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.949761] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.951917] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.954393] PCI: CLS 0 bytes, default 64 [ 0.955784] Unpacking initramfs... [ 2.397510] debug: unmapping init [mem 0xffff89077cc54000-0xffff89077ffbffff] [ 2.405060] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.407502] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.412200] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.936947] Initialise system trusted keyrings [ 2.938753] Key type blacklist registered [ 2.940617] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.951194] zbud: loaded [ 2.954397] *** VALIDATE nfs *** [ 2.955873] *** VALIDATE nfs4 *** [ 2.957781] pstore: using deflate compression [ 2.961084] Platform Keyring initialized [ 3.072857] NET: Registered protocol family 38 [ 3.074365] Key type asymmetric registered [ 3.076056] Asymmetric key parser 'x509' registered [ 3.078512] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.082491] io scheduler mq-deadline registered [ 3.084370] io scheduler kyber registered [ 3.086261] io scheduler bfq registered [ 3.088647] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.091472] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.094701] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.097104] ACPI: Power Button [PWRF] [ 3.101486] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.108852] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.123145] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.133104] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.147479] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.176776] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.204777] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.209772] Non-volatile memory driver v1.3 [ 3.211301] Linux agpgart interface v0.103 [ 3.242957] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.246108] vda: detected capacity change from 0 to 75194368 [ 3.259841] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.263105] vdb: detected capacity change from 0 to 1073741824 [ 3.278524] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.281490] vdc: detected capacity change from 0 to 2621440000 [ 3.300792] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.304551] vdd: detected capacity change from 0 to 2621440000 [ 3.319986] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322790] vde: detected capacity change from 0 to 4294967296 [ 3.337290] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.341182] vdf: detected capacity change from 0 to 4294967296 [ 3.346895] libphy: Fixed MDIO Bus: probed [ 3.351563] usbcore: registered new interface driver usbserial_generic [ 3.354106] usbserial: USB Serial support registered for generic [ 3.356365] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.361157] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.362814] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.365541] mousedev: PS/2 mouse device common for all mice [ 3.368314] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.370553] rtc_cmos 00:05: RTC can wake from S4 [ 3.376764] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.377974] rtc_cmos 00:05: registered as rtc0 [ 3.382158] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.384465] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.386298] intel_pstate: CPU model not supported [ 3.392383] hid: raw HID events driver (C) Jiri Kosina [ 3.394856] usbcore: registered new interface driver usbhid [ 3.397779] usbhid: USB HID core driver [ 3.399830] drop_monitor: Initializing network drop monitor service [ 3.402467] Initializing XFRM netlink socket [ 3.404763] NET: Registered protocol family 10 [ 3.408033] Segment Routing with IPv6 [ 3.409653] NET: Registered protocol family 17 [ 3.411775] mpls_gso: MPLS GSO support [ 3.417203] RAS: Correctable Errors collector initialized. [ 3.419600] AVX version of gcm_enc/dec engaged. [ 3.421288] AES CTR mode by8 optimization enabled [ 3.519140] sched_clock: Marking stable (3519109846, 0)->(4506998065, -987888219) [ 3.522978] registered taskstats version 1 [ 3.525293] Loading compiled-in X.509 certificates [ 3.527177] zswap: loaded using pool lzo/zbud [ 3.555912] Key type big_key registered [ 3.570267] Key type encrypted registered [ 3.572112] ima: No TPM chip found, activating TPM-bypass! [ 3.574357] ima: Allocated hash algorithm: sha1 [ 3.576329] ima: No architecture policies found [ 3.578274] evm: Initialising EVM extended attributes: [ 3.580390] evm: security.selinux [ 3.581844] evm: security.ima [ 3.583106] evm: security.capability [ 3.584538] evm: HMAC attrs: 0x1 [ 3.587244] rtc_cmos 00:05: setting system clock to 2026-08-27 18:08:12 UTC (1787854092) [ 3.593712] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.596972] debug: unmapping init [mem 0xffffffff89982000-0xffffffff89c58fff] [ 3.606169] Write protecting the kernel read-only data: 28672k [ 3.610092] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.612911] debug: unmapping init [mem 0xffffffff88914000-0xffffffff889fffff] [ 3.652752] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.662546] systemd[1]: Detected virtualization kvm. [ 3.664496] systemd[1]: Detected architecture x86-64. [ 3.667993] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.693266] systemd[1]: No hostname configured. [ 3.695135] systemd[1]: Set hostname to . [ 3.697337] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699978] systemd[1]: Initializing machine ID from random generator. [ 3.856212] random: systemd: uninitialized urandom read (16 bytes read) [ 3.859479] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.864421] random: systemd: uninitialized urandom read (16 bytes read) [ 3.868037] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.873031] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ 4.147717] random: fast init done [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.531890] device-mapper: uevent: version 1.0.3 [ 4.533837] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.252378] virtio_net virtio0 ens2: renamed from eth0 [ 5.363196] scsi host0: ata_piix [ 5.378728] scsi host1: ata_piix [ 5.382170] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.385226] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.816895] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.098436] random: crng init done [ 10.099904] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.600324] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.769775] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.022685] SELinux: Disabled at runtime. [ 12.090700] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.100366] systemd[1]: Detected virtualization kvm. [ 12.102541] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.591920] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.595506] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.600950] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.606288] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.609899] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.617316] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.624764] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK [0[ 12.669948] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.032324] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.289855] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.314462] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.439757] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.456773] EDAC sbridge: Ver: 1.1.2 [ 14.983600] Key type dns_resolver registered [ 15.289192] NFS: Registering the id_resolver key type [ 15.291345] Key type id_resolver registered [ 15.293020] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg629-server login: [ 41.699202] hrtimer: interrupt took 11340215 ns [ 55.564664] libcfs: loading out-of-tree module taints kernel. [ 55.609175] Key type ._llcrypt registered [ 55.610632] Key type .llcrypt registered [ 55.735651] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 76.980450] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 78.761067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.803394] alg: No test for adler32 (adler32-zlib) [ 80.498896] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 81.530762] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 83.224569] Key type lgssc registered [ 85.117413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.227122] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.015472] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 162.362540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.390875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.672251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.699166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.802546] Lustre: lustre-MDT0000: new disk, initializing [ 163.890078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.913464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.125383] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.842840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.040406] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.107528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.116424] Lustre: Skipped 1 previous similar message [ 184.259181] Lustre: lustre-MDT0001: new disk, initializing [ 184.353390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.397864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.416037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.885502] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.990675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.628845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.870594] Lustre: lustre-OST0000: new disk, initializing [ 204.874982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.880760] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.949603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.716062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.731145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.784050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.534989] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.137067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.243237] Lustre: lustre-OST0001: new disk, initializing [ 226.247654] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.255777] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.329545] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.048375] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.517613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.527104] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.584792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.271321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.029635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 262.206711] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing check_logdir /tmp/testlogs/ [ 267.986274] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing yml_node [ 272.886574] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 275.088904] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 277.313905] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 279.145728] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 14:12:46 EDT 2026 [ 295.527875] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 297.047092] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.389980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.392578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.405097] Lustre: Skipped 2 previous similar messages [ 316.446319] Lustre: Skipped 3 previous similar messages [ 319.511430] Lustre: server umount lustre-MDT0000 complete [ 323.489413] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854412 with bad export cookie 17557683634797186999 [ 323.490954] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.496609] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 326.626202] LustreError: 6508:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.627485] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.630263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.657046] LustreError: 6508:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 326.689856] Lustre: Skipped 2 previous similar messages [ 328.675078] LustreError: 6510:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.675886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.690744] LustreError: 6510:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 328.700566] Lustre: Skipped 2 previous similar messages [ 329.914597] Lustre: server umount lustre-MDT0001 complete [ 333.786907] Lustre: server umount lustre-OST0000 complete [ 337.312300] Lustre: server umount lustre-OST0001 complete [ 343.493442] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 353.134980] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 389.693598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.921695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.967950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.096240] Lustre: lustre-MDT0000: new disk, initializing [ 390.218626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.255526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 394.531205] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 403.763283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.889058] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.944260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.948148] Lustre: Skipped 1 previous similar message [ 404.048979] Lustre: lustre-MDT0001: new disk, initializing [ 404.142466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.190501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.203866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.210473] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 417.759725] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.448988] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 430.210423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 430.513376] Lustre: lustre-OST0000: new disk, initializing [ 430.518600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 430.524391] Lustre: 17032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 430.613065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.032325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 432.042923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 432.147266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 436.449757] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 445.807699] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 450.131799] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 450.342761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.553679] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.722109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.698967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.701645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.703644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.745578] Lustre: Skipped 1 previous similar message [ 462.765950] Lustre: server umount lustre-OST0000 complete [ 467.942097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.945793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.962397] Lustre: Skipped 1 previous similar message [ 467.965571] Lustre: Skipped 1 previous similar message [ 477.158291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.168326] Lustre: Skipped 4 previous similar messages [ 481.248125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 481.407749] Lustre: server umount lustre-MDT0000 complete [ 482.273617] LustreError: 15077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.295264] LustreError: 15077:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 484.718567] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854573 with bad export cookie 17557683634797188532 [ 484.720562] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.734366] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 484.952338] Lustre: server umount lustre-MDT0001 complete [ 488.711270] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:16:15 (1787854575) [ 491.089593] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 492.678930] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:16:19 (1787854579) [ 498.781568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.364302] LustreError: 19441:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.452330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.716144] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 504.804980] LustreError: 19442:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.114949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.287970] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 524.426351] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.918434] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.051855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.840440] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 547.565143] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 558.049145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.055946] LustreError: Skipped 1 previous similar message [ 558.058769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.069809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.097795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 561.350115] Lustre: server umount lustre-OST0000 complete [ 570.795630] Lustre: server umount lustre-MDT0000 complete [ 574.261426] LustreError: 19423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854663 with bad export cookie 17557683634797189414 [ 574.264140] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.272600] LustreError: 19423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 574.716108] Lustre: server umount lustre-MDT0001 complete [ 585.367424] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 587.988650] Key type lgssc unregistered [ 588.271222] LNet: 22974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.286913] LNetError: 22974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.315844] LNet: Removed LNI 192.168.206.129@tcp [ 589.713171] Key type .llcrypt unregistered [ 589.717160] Key type ._llcrypt unregistered [ 613.871910] Key type ._llcrypt registered [ 613.875630] Key type .llcrypt registered [ 613.940902] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 614.774279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 615.006175] alg: No test for adler32 (adler32-zlib) [ 616.200708] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 616.602756] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 618.393062] Key type lgssc registered [ 619.506449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.786498] loop: module loaded [ 628.861636] loop0: detected capacity change from 0 to 204800000 [ 629.202155] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 629.217759] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 629.554741] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.513664] loop0: detected capacity change from 0 to 204800000 [ 635.721688] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.732602] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.752016] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.893910] loop0: detected capacity change from 0 to 204800000 [ 646.196955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 646.249291] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.562337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 647.642860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 647.735398] Lustre: lustre-MDT0000: new disk, initializing [ 647.832261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.864724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 652.078771] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 660.377364] loop1: detected capacity change from 0 to 204800000 [ 665.860626] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.164110] Lustre: lustre-OST07c6: new disk, initializing [ 666.172778] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 666.181460] Lustre: Skipped 1 previous similar message [ 666.193949] Lustre: 26161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 666.278554] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 672.987968] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 674.993241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 675.019422] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 675.075649] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 678.777676] Lustre: 25145:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 678.786531] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 678.795594] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 678.807104] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 678.816981] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 678.826571] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 692.019754] Lustre: 25147:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 692.027974] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 692.036419] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 692.046199] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 692.065345] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 692.075134] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.932709] Lustre: 25147:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 696.941172] Lustre: 25147:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 696.946354] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 696.951556] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.956689] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.961843] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.967138] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 696.973223] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.978126] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.983676] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.988864] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.995696] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.140096] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 700.149052] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 700.155552] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 700.166562] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.177897] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 700.183409] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.189538] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 700.198579] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.205881] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 700.211137] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.218600] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.224524] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.034412] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 705.045855] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 705.058630] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 705.071797] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.082521] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 705.090033] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.098883] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 705.112137] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.126873] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 705.135924] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.142705] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.147716] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.272873] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 716.285440] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 716.300022] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 716.310398] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.317312] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.324501] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.337671] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 716.345436] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.360186] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 716.373502] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.384516] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.392538] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 767.457244] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 767.467037] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.483760] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 770.475123] Lustre: server umount lustre-OST07c6 complete [ 778.385976] Lustre: server umount lustre-MDT0000 complete [ 793.063595] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 795.470799] Key type lgssc unregistered [ 795.690870] LNet: 28721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.698224] LNetError: 28721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.728230] LNet: Removed LNI 192.168.206.129@tcp [ 796.451134] Key type .llcrypt unregistered [ 796.453731] Key type ._llcrypt unregistered [ 811.881254] Key type ._llcrypt registered [ 811.883623] Key type .llcrypt registered [ 811.985783] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 812.595742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.607307] alg: No test for adler32 (adler32-zlib) [ 813.575330] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 813.829263] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 815.464700] Key type lgssc registered [ 816.361461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 823.518308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 823.549828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.062288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.676653] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 834.534359] LustreError: 29988:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.552923] LustreError: 29988:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 838.945433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.459872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 844.339252] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 852.498737] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 858.233518] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 864.838652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 865.290268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.368746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 870.001483] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 876.694856] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.064990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.068105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.074752] LustreError: Skipped 1 previous similar message [ 900.075578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.084228] Lustre: Skipped 1 previous similar message [ 900.101101] Lustre: Skipped 1 previous similar message [ 905.189580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.194805] Lustre: Skipped 1 previous similar message [ 906.184401] Lustre: server umount lustre-OST0000 complete [ 910.306985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.320455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.325478] Lustre: Skipped 1 previous similar message [ 910.343121] Lustre: Skipped 1 previous similar message [ 915.429183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 915.437347] Lustre: Skipped 1 previous similar message [ 915.817404] Lustre: server umount lustre-MDT0000 complete [ 919.095559] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855008 with bad export cookie 1885905028271442135 [ 919.098051] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.113427] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 919.588372] Lustre: server umount lustre-MDT0001 complete [ 928.862358] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 931.117459] Key type lgssc unregistered [ 931.375868] LNet: 33750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.384946] LNetError: 33750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.405534] LNet: Removed LNI 192.168.206.129@tcp [ 932.046896] Key type .llcrypt unregistered [ 932.049775] Key type ._llcrypt unregistered [ 948.536899] Key type ._llcrypt registered [ 948.539283] Key type .llcrypt registered [ 948.683908] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 949.724867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.872221] alg: No test for adler32 (adler32-zlib) [ 950.862527] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 951.093469] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 952.761481] Key type lgssc registered [ 953.849277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.091706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.107615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.480412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.682419] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 971.749264] LustreError: 35019:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.777087] LustreError: 35019:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 975.294172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.848228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 981.730789] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 989.790745] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 996.938777] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1004.213062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.689932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.801834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1010.583927] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1017.647825] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1033.697883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.705075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.725138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.802211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.805809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.815431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.512517] Lustre: server umount lustre-OST0000 complete [ 1042.402213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1042.414796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.435182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.443300] Lustre: Skipped 1 previous similar message [ 1047.011189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.014372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.725977] Lustre: server umount lustre-MDT0000 complete [ 1052.129669] LustreError: 35018:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.151207] LustreError: 35018:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1053.257118] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855142 with bad export cookie 1873860498419375978 [ 1053.264656] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.273548] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1053.760659] Lustre: server umount lustre-MDT0001 complete [ 1064.449176] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 1067.156540] Key type lgssc unregistered [ 1067.426299] LNet: 38582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.433962] LNetError: 38582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.457505] LNet: Removed LNI 192.168.206.129@tcp [ 1068.187338] Key type .llcrypt unregistered [ 1068.189435] Key type ._llcrypt unregistered [ 1091.020503] Key type ._llcrypt registered [ 1091.022436] Key type .llcrypt registered [ 1091.147354] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 1092.511606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1092.547145] alg: No test for adler32 (adler32-zlib) [ 1093.664926] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1093.930660] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 1095.568175] Key type lgssc registered [ 1096.721859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1103.958518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1103.985796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.534759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.853195] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1115.116653] LustreError: 40000:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.141308] LustreError: 40000:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1117.806397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.226826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.795345] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1130.452383] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1135.757654] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1141.716527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.136078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.581678] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1150.393672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1154.560259] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1162.083769] Lustre: server umount lustre-MDT0000 complete [ 1162.235431] LustreError: 39996:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.260119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1164.279926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.299556] LustreError: 40000:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.795110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.808025] Lustre: Skipped 1 previous similar message [ 1166.234021] LustreError: 39982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855255 with bad export cookie 2607094997957128036 [ 1166.239430] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.246293] LustreError: 39982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.280221] LustreError: 39996:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.302092] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.29@tcp (stopping) [ 1167.302898] LustreError: 39996:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1170.913498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.928313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1171.938616] LustreError: 41753:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.939513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1171.968165] LustreError: 41753:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1172.784307] Lustre: server umount lustre-MDT0001 complete [ 1193.699891] Lustre: server umount lustre-OST0000 complete [ 1207.378672] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 1210.009154] Key type lgssc unregistered [ 1210.239461] LNet: 43860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.250680] LNetError: 43860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.264750] LNet: Removed LNI 192.168.206.129@tcp [ 1210.956886] Key type .llcrypt unregistered [ 1210.958922] Key type ._llcrypt unregistered [ 1232.479165] Key type ._llcrypt registered [ 1232.481548] Key type .llcrypt registered [ 1232.593163] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 1233.784827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1233.811961] alg: No test for adler32 (adler32-zlib) [ 1234.881790] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1235.196807] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 1236.880480] Key type lgssc registered [ 1237.676995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1244.543580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.559783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.926088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.317671] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1255.407276] LustreError: 45275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.426682] LustreError: 45275:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1256.856095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.370358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1262.056120] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1271.077213] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1277.457468] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1283.660318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.974384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.131853] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1291.124225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1296.478703] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1307.619827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.643585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.644216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.990979] Lustre: server umount lustre-OST0000 complete [ 1326.255121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.277167] Lustre: Skipped 1 previous similar message [ 1326.284740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.290894] Lustre: Skipped 2 previous similar messages [ 1327.331438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.335388] Lustre: Skipped 1 previous similar message [ 1332.691884] Lustre: server umount lustre-MDT0000 complete [ 1332.708777] LustreError: 46003:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.725554] LustreError: 46003:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1336.359866] LustreError: 47972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855425 with bad export cookie 9950411461128615734 [ 1336.361088] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.369825] LustreError: 47972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1342.806927] Lustre: server umount lustre-MDT0001 complete [ 1354.965577] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 1358.545749] Key type lgssc unregistered [ 1359.013589] LNet: 48987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.025103] LNetError: 48987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.042353] LNet: Removed LNI 192.168.206.129@tcp [ 1360.050128] Key type .llcrypt unregistered [ 1360.054662] Key type ._llcrypt unregistered [ 1381.407524] Key type ._llcrypt registered [ 1381.412830] Key type .llcrypt registered [ 1381.519489] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 1382.208250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.248439] alg: No test for adler32 (adler32-zlib) [ 1383.262884] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1383.524252] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 1385.208151] Key type lgssc registered [ 1386.137310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.728918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1392.759831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.090732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.996806] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1403.384716] LustreError: 50382:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.411633] LustreError: 50382:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1407.420118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.640891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.200920] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1421.798449] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1427.120463] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1433.982063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.501371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.657777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1441.456555] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1448.912884] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1458.479633] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1461.954783] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1474.314776] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2714h32m40s [ 1476.268308] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1482.708205] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.29@tcp (stopping) [ 1483.748272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.752896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.765086] Lustre: Skipped 1 previous similar message [ 1483.773304] Lustre: Skipped 1 previous similar message [ 1484.834781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1484.860543] Lustre: Skipped 4 previous similar messages [ 1485.874678] Lustre: server umount lustre-MDT0000 complete [ 1489.452307] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1489.856376] LustreError: 50377:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.917080] LustreError: 50381:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.929266] LustreError: 50381:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1495.979086] LustreError: 50376:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.298409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.499704] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1496.835966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.920235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1497.840060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1501.682857] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1505.380979] Lustre: DEBUG MARKER: Wait for df (36711) ... [ 1506.802185] Lustre: DEBUG MARKER: done [ 1512.417960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.430736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.458194] Lustre: Skipped 1 previous similar message [ 1512.470103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.442297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.573732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1518.586285] Lustre: Skipped 3 previous similar messages [ 1525.217113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1525.548032] Lustre: server umount lustre-OST0000 complete [ 1533.921803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1533.925497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.938766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.938784] Lustre: Skipped 2 previous similar messages [ 1533.988344] Lustre: Skipped 1 previous similar message [ 1535.753570] Lustre: server umount lustre-MDT0000 complete [ 1539.236475] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855628 with bad export cookie 91778619732339394 [ 1539.240405] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.248770] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1539.584916] Lustre: server umount lustre-MDT0001 complete [ 1551.378705] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 1554.372316] Key type lgssc unregistered [ 1554.772711] LNet: 55309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.783657] LNetError: 55309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.799304] LNet: Removed LNI 192.168.206.129@tcp [ 1556.061211] Key type .llcrypt unregistered [ 1556.067539] Key type ._llcrypt unregistered [ 1595.865488] Key type ._llcrypt registered [ 1595.867649] Key type .llcrypt registered [ 1595.990302] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 1597.148549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.328361] alg: No test for adler32 (adler32-zlib) [ 1598.427845] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1598.652614] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 1600.352166] Key type lgssc registered [ 1601.343624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1608.709832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1608.733588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.049643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1610.079935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1610.289702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.547994] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1623.928194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.096937] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1624.198049] Lustre: 57519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1624.571467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.635284] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1637.981643] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1644.783749] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1651.650122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.856665] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1652.160708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.006715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1658.956437] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1667.230988] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1674.916717] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2714h36m01s [ 1676.368981] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1683.944382] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1686.760919] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1686.816368] LustreError: 58783:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1706.967898] Lustre: lustre-MDT0000: Client a36a5d25-2a8a-4259-a416-fcefb0b53e2a (at 192.168.206.29@tcp) reconnecting [ 1726.912350] LustreError: 58783:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1729.158883] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1731.565474] Lustre: DEBUG MARKER: done [ 1739.745375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1739.751622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.772574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.274619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.425936] Lustre: server umount lustre-OST0000 complete [ 1745.377845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.380210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.413445] Lustre: Skipped 1 previous similar message [ 1745.457241] Lustre: Skipped 1 previous similar message [ 1746.533767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.549396] Lustre: Skipped 1 previous similar message [ 1750.808871] Lustre: server umount lustre-MDT0000 complete [ 1751.604743] LustreError: 60100:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.660634] LustreError: 60100:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 1755.325433] LustreError: 57512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855844 with bad export cookie 16093860787940223942 [ 1755.335655] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1755.348649] LustreError: 57512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1756.045159] Lustre: server umount lustre-MDT0001 complete [ 1769.379404] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 1772.422933] Key type lgssc unregistered [ 1772.798356] LNet: 61851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.815202] LNetError: 61851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.854120] LNet: Removed LNI 192.168.206.129@tcp [ 1773.650192] Key type .llcrypt unregistered [ 1773.656433] Key type ._llcrypt unregistered [ 1816.088945] Key type ._llcrypt registered [ 1816.091123] Key type .llcrypt registered [ 1816.192862] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 1817.188352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1817.271550] alg: No test for adler32 (adler32-zlib) [ 1818.330452] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 1818.533660] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 1820.216313] Key type lgssc registered [ 1821.353564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1825.163143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1825.174534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1826.440497] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1826.482629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1826.655861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.557889] Lustre: Failing over lustre-MDT0000 [ 1827.929553] Lustre: server umount lustre-MDT0000 complete [ 1835.204614] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:38:42 (1787855922) [ 1841.309333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.756977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.128843] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1853.740946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.811777] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1853.848314] Lustre: 64650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1854.098991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1858.315978] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1865.150061] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1869.965942] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1875.350054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.501687] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1875.650653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.865118] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1884.798710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1887.933656] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1895.182775] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1905.123297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.141679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.168977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.653891] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.672849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.698712] Lustre: Skipped 1 previous similar message [ 1910.759792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.767813] Lustre: Skipped 3 previous similar messages [ 1915.875469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.887863] Lustre: Skipped 2 previous similar messages [ 1916.384564] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.533279] Lustre: server umount lustre-MDT0000 complete [ 1920.329914] LustreError: 64643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856009 with bad export cookie 5298588156237045462 [ 1920.336425] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.346404] LustreError: 64643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1920.655241] Lustre: server umount lustre-MDT0001 complete [ 1923.929827] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1928.198602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1936.353357] Lustre: 63573:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856009/real 1787856009] req@000000008eb5aa55 x1874702795578752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1936.397926] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.110927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.496517] Lustre: 63574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856015/real 1787856015] req@00000000699cd58c x1874702795578880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856031 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1945.633796] LustreError: 63571:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009e8fce9d x1874702795579520/t0(0) o250->MGC192.168.206.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1946.190554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.295073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1947.191197] Lustre: 63574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856019/real 1787856019] req@0000000070a50f34 x1874702795579136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1950.834548] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1951.219121] Lustre: 63573:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856024/real 1787856024] req@000000007dbea38b x1874702795579264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1955.304466] LustreError: 68420:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.324629] LustreError: 68420:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 1959.265591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.671672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.706972] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 1965.038401] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1970.219182] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1973.599567] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.227969] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.069645] Lustre: server umount lustre-MDT0000 complete [ 1990.012747] LustreError: 64641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856078 with bad export cookie 5298588156237047737 [ 1990.014976] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.028720] LustreError: 64641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1990.424393] Lustre: server umount lustre-MDT0001 complete [ 1997.395760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2006.496127] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856079/real 1787856079] req@000000009152f72c x1874702795604736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2006.519183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.521545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.757812] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49885ddadf99ed1c [ 2014.779549] Lustre: MGC192.168.206.129@tcp: Connection restored to 0@lo (at 0@lo) [ 2015.079041] LustreError: 66497:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.166750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.274281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2016.164264] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856088/real 1787856088] req@0000000039afa464 x1874702795605376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2016.169146] LustreError: 71103:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.229342] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2016.247591] LustreError: 71103:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2020.186640] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2021.363680] LustreError: 71104:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.401855] LustreError: 71104:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2026.472756] LustreError: 71439:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.484673] LustreError: 71439:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2029.198053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.672967] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2034.685968] Lustre: Skipped 1 previous similar message [ 2035.539188] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2045.206320] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.957808] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2060.267045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2060.276386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.286157] LustreError: 71439:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.340887] Lustre: Skipped 3 previous similar messages [ 2060.387954] Lustre: server umount lustre-MDT0000 complete [ 2064.147591] LustreError: 64643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856153 with bad export cookie 5298588156237049116 [ 2064.154205] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.162783] LustreError: 64643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2064.468253] Lustre: server umount lustre-MDT0001 complete [ 2072.464570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2076.697130] Lustre: DEBUG MARKER: files compared the same [ 2088.369302] Lustre: server umount lustre-OST0000 complete [ 2104.897497] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 2108.030887] Key type lgssc unregistered [ 2108.351909] LNet: 74382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.360796] LNetError: 74382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.381905] LNet: Removed LNI 192.168.206.129@tcp [ 2109.423239] Key type .llcrypt unregistered [ 2109.424847] Key type ._llcrypt unregistered [ 2135.891753] Key type ._llcrypt registered [ 2135.895639] Key type .llcrypt registered [ 2136.012622] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 2137.307520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.390173] alg: No test for adler32 (adler32-zlib) [ 2138.517969] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2138.817215] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 2140.456390] Key type lgssc registered [ 2141.927281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.897680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.943500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.608586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.974495] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug +malloc all [ 2162.146945] LustreError: 75802:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.175572] LustreError: 75802:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2166.547825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.857654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.809827] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug +malloc all [ 2178.940710] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2184.177194] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.134417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.412546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.964463] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug +malloc all [ 2197.606580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2204.406524] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2212.326995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.333342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.348538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2213.347784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2213.348820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.380583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.476125] LustreError: 77624:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.503766] LustreError: 77624:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2218.718182] Lustre: server umount lustre-OST0000 complete [ 2223.073029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2223.082769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.100919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.113734] Lustre: Skipped 1 previous similar message [ 2228.705812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.713669] Lustre: Skipped 2 previous similar messages [ 2229.206448] Lustre: server umount lustre-MDT0000 complete [ 2232.903664] LustreError: 75783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856321 with bad export cookie 16912080445180151743 [ 2232.919599] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.355343] Lustre: server umount lustre-MDT0001 complete [ 2243.731861] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 2246.305766] Key type lgssc unregistered [ 2246.606803] LNet: 79337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.614484] LNetError: 79337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2246.628181] LNet: Removed LNI 192.168.206.129@tcp [ 2247.454194] Key type .llcrypt unregistered [ 2247.458362] Key type ._llcrypt unregistered [ 2269.754726] Key type ._llcrypt registered [ 2269.758318] Key type .llcrypt registered [ 2269.871145] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 2270.997210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.043331] alg: No test for adler32 (adler32-zlib) [ 2272.129675] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2272.492855] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 2274.208161] Key type lgssc registered [ 2275.336720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.874309] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.888533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.493911] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2304.609976] LustreError: 80708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.129@tcp: failed processing log, type 4: rc = -110 [ 2331.360829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.255061] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2344.366578] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2351.358935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.524320] Lustre: *** cfs_fail_loc=706, val=0*** [ 2351.903367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.968141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2356.444563] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2357.220035] LustreError: 80733:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.253756] LustreError: 80733:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2362.346496] LustreError: 80734:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.309809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.703727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2369.721561] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2377.507808] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.830292] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.380307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.386183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.401139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.427783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.452189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.550387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.556934] Lustre: Skipped 1 previous similar message [ 2397.669056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.678263] Lustre: Skipped 1 previous similar message [ 2399.200175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2399.388863] Lustre: server umount lustre-OST0000 complete [ 2405.869182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.880328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.907350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.917858] Lustre: Skipped 1 previous similar message [ 2407.908670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.631475] Lustre: server umount lustre-MDT0000 complete [ 2413.025465] LustreError: 81792:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.060869] LustreError: 81792:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2414.077909] LustreError: 80714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856502 with bad export cookie 10166036817604463708 [ 2414.087961] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.568039] Lustre: server umount lustre-MDT0001 complete [ 2427.742569] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 2431.365933] Key type lgssc unregistered [ 2431.753778] LNet: 84326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.764110] LNetError: 84326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.816253] LNet: Removed LNI 192.168.206.129@tcp [ 2434.446915] Key type .llcrypt unregistered [ 2434.450065] Key type ._llcrypt unregistered [ 2459.928509] Key type ._llcrypt registered [ 2459.931409] Key type .llcrypt registered [ 2460.059690] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 2461.405108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.436185] alg: No test for adler32 (adler32-zlib) [ 2462.510231] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2462.681468] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 2464.336150] Key type lgssc registered [ 2465.555696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.457154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.484360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.034513] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2488.036594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.461587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2493.037534] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2493.921412] LustreError: 86434:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.943781] LustreError: 86434:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2499.042714] LustreError: 86433:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.068765] LustreError: 86433:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2501.696233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.215201] LustreError: 86437:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.250412] LustreError: 86437:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2502.258607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.233812] LustreError: 86438:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.250448] LustreError: 86438:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2508.132184] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2512.380429] LustreError: 86437:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.409229] LustreError: 86437:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2516.505305] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.519804] Lustre: MGS: Client bd785f6b-7e02-44af-9674-9d1b72610528 (at 0@lo) reconnecting [ 2516.546671] LustreError: 86448:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2516.784627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.859993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2520.646940] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2534.369701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.381701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.412797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.442472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.445672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2537.458425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.052572] Lustre: server umount lustre-OST0000 complete [ 2547.682535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.682860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.693214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.701541] Lustre: Skipped 1 previous similar message [ 2547.714388] Lustre: Skipped 1 previous similar message [ 2549.884248] Lustre: server umount lustre-MDT0000 complete [ 2552.807216] LustreError: 86434:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.857682] LustreError: 86434:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2554.356459] Lustre: server umount lustre-MDT0001 complete [ 2557.581401] Lustre: server umount MGS complete [ 2569.843052] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:50:56 (1787856656) [ 2596.627298] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 2608.246370] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 2646.423923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.272403] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2659.053547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.303761] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2659.335798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2659.409602] Lustre: lustre-MDT0001: new disk, initializing [ 2659.481786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2659.511068] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2659.519955] Lustre: 93603:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2659.532276] LustreError: 93603:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2663.377191] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2673.168920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.481855] Lustre: lustre-OST0000: new disk, initializing [ 2673.486506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2673.494546] Lustre: 94371:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2673.505419] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2673.511982] Lustre: Skipped 1 previous similar message [ 2673.567325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.837378] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2679.777312] Lustre: 94374:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000032ad9175 x1874703470861952/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787856784 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2688.359878] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2695.601377] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.635235] Lustre: MGS: Client b5bacc55-9481-4dbf-9796-5fa412a667f1 (at 0@lo) reconnecting [ 2695.673577] LustreError: 93589:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2695.721941] Lustre: 95415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2695.783511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2695.901255] Lustre: lustre-MDT0000: new disk, initializing [ 2696.047380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.066354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2696.085915] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2697.125922] Lustre: 95440:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bffa1489 x1874703470869504/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787856802 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2700.750851] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2702.240696] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2706.221245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2706.237268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2706.313517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2732.009077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.016176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.046196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2736.424812] Lustre: server umount lustre-OST0000 complete [ 2744.288948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2744.291183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.300808] LustreError: Skipped 1 previous similar message [ 2744.324299] Lustre: Skipped 1 previous similar message [ 2744.327832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.327838] Lustre: Skipped 3 previous similar messages [ 2746.049879] Lustre: server umount lustre-MDT0000 complete [ 2755.743469] Lustre: server umount lustre-MDT0001 complete [ 2759.664087] Lustre: server umount MGS complete [ 2771.421230] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:54:18 (1787856858) [ 2790.559186] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 2793.474921] Key type lgssc unregistered [ 2793.767570] LNet: 97942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2793.782092] LNetError: 97942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2793.804802] LNet: Removed LNI 192.168.206.129@tcp [ 2794.849798] Key type .llcrypt unregistered [ 2794.852445] Key type ._llcrypt unregistered [ 2811.478855] Key type ._llcrypt registered [ 2811.483210] Key type .llcrypt registered [ 2811.593300] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 2812.844529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2812.852405] alg: No test for adler32 (adler32-zlib) [ 2813.894842] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2814.189978] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 2815.928146] Key type lgssc registered [ 2817.164755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.956500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2823.983140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.390732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.083826] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2835.939242] LustreError: 99163:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.957419] LustreError: 99163:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2837.042036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.401108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2841.561357] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2849.203649] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2854.307545] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2861.183379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2861.511581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.045986] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2870.702736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2875.365260] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2883.042135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2883.052590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.067780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.951758] Lustre: server umount lustre-OST0000 complete [ 2888.674735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2888.685401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.694672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.234657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.236082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.353885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.363983] Lustre: Skipped 2 previous similar messages [ 2897.488193] Lustre: server umount lustre-MDT0000 complete [ 2900.858809] LustreError: 99144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856989 with bad export cookie 903528361670253340 [ 2900.867739] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.871640] LustreError: 99144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2901.268745] Lustre: server umount lustre-MDT0001 complete [ 2910.923565] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 2913.331723] Key type lgssc unregistered [ 2913.579573] LNet: 102697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2913.588081] LNetError: 102697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2913.604809] LNet: Removed LNI 192.168.206.129@tcp [ 2914.277895] Key type .llcrypt unregistered [ 2914.279909] Key type ._llcrypt unregistered [ 2929.030281] Key type ._llcrypt registered [ 2929.033236] Key type .llcrypt registered [ 2929.125609] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 2930.084912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2930.266278] alg: No test for adler32 (adler32-zlib) [ 2931.316513] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 2931.564820] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 2933.264469] Key type lgssc registered [ 2934.298984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2940.382836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2940.391192] LustreError: 103780:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2940.403134] LustreError: 103781:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2940.412418] LustreError: 103780:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 2940.421369] LustreError: 103780:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2940.429064] LustreError: 103780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2940.442292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.894734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.797312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.039466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.398870] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 2958.531627] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing lsmod [ 2961.388478] LustreError: 104984:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2961.473956] LustreError: 104988:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2961.500275] LustreError: 104984:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4911 [ 2961.510263] LustreError: 104984:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2961.525575] LustreError: 104984:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2961.527892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2961.971193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.012341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.026270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.045320] Lustre: Skipped 1 previous similar message [ 2967.053126] Lustre: Skipped 1 previous similar message [ 2970.876187] Lustre: server umount lustre-MDT0000 complete [ 2972.138602] LustreError: 105174:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.180214] LustreError: 105174:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2973.172823] LustreError: 105174:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.200396] LustreError: 105174:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2975.015362] LustreError: 103798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857063 with bad export cookie 5506612593487208197 [ 2975.019933] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.026522] LustreError: 103798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2975.280261] LustreError: 104249:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a2e3928a x1874703962624128/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2975.436725] Lustre: server umount lustre-MDT0001 complete [ 2990.206666] Lustre: server umount lustre-OST0000 complete [ 2999.436723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.017593] LustreError: 106052:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.165236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.189292] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3005.411655] LustreError: 106053:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.507298] LustreError: 106052:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.302662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.632907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3018.477739] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3025.807782] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3031.595773] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3037.994960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.181366] LustreError: 107872:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.455344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.813821] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3044.859591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3052.277239] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3059.173436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.182703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.200879] Lustre: Skipped 1 previous similar message [ 3059.213407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.222893] Lustre: Skipped 1 previous similar message [ 3060.202040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3060.223330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.249493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.253922] Lustre: Skipped 1 previous similar message [ 3063.869987] Lustre: server umount lustre-OST0000 complete [ 3069.923268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3069.932841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.953097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3073.752879] Lustre: server umount lustre-MDT0000 complete [ 3075.553077] LustreError: 107925:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.576287] LustreError: 107925:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3077.073627] LustreError: 106034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857165 with bad export cookie 5506612593487208897 [ 3077.088102] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.484840] Lustre: server umount lustre-MDT0001 complete [ 3088.455293] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 3091.141592] Key type lgssc unregistered [ 3091.382649] LNet: 109589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3091.397481] LNetError: 109589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3091.419311] LNet: Removed LNI 192.168.206.129@tcp [ 3092.294627] Key type .llcrypt unregistered [ 3092.298226] Key type ._llcrypt unregistered [ 3113.786130] Key type ._llcrypt registered [ 3113.788063] Key type .llcrypt registered [ 3113.906356] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 3114.889319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3114.942085] alg: No test for adler32 (adler32-zlib) [ 3115.982436] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3116.234349] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 3117.968203] Key type lgssc registered [ 3119.423337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3127.932986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3127.948855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.616382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.986487] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3140.066107] LustreError: 110999:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.099346] LustreError: 110999:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3142.194354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.521127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3146.379136] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3153.519842] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3158.690813] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3164.036647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3164.293359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.947963] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3174.577846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3176.337647] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3181.427598] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3184.229366] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3188.195992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3188.204133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.222628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.249141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.255242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.256038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.280522] Lustre: Skipped 1 previous similar message [ 3193.459337] Lustre: server umount lustre-OST0000 complete [ 3198.946353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3198.962523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.996503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3202.927630] Lustre: server umount lustre-MDT0000 complete [ 3205.601971] LustreError: 112053:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.611882] LustreError: 112053:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3206.333799] LustreError: 113543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857295 with bad export cookie 15318329941715531879 [ 3206.334227] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3206.354673] LustreError: 113543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3206.615423] Lustre: server umount lustre-MDT0001 complete [ 3216.187198] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 3218.546653] Key type lgssc unregistered [ 3218.831220] LNet: 114651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.840976] LNetError: 114651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3218.867422] LNet: Removed LNI 192.168.206.129@tcp [ 3219.773171] Key type .llcrypt unregistered [ 3219.776772] Key type ._llcrypt unregistered [ 3239.049465] Key type ._llcrypt registered [ 3239.054083] Key type .llcrypt registered [ 3239.201228] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 3239.991146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3240.017231] alg: No test for adler32 (adler32-zlib) [ 3241.188920] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3241.408265] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 3243.072222] Key type lgssc registered [ 3244.149660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3251.658711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3251.679251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.170295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.040672] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3263.463505] LustreError: 115875:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.480327] LustreError: 115875:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3264.709291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.074455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3269.268649] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3276.048434] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3281.860515] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3289.505125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.573604] LustreError: 117694:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.778831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.029628] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3298.299136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3303.899801] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3313.633476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.640066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.650966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3313.681114] Lustre: Skipped 1 previous similar message [ 3318.263953] Lustre: server umount lustre-OST0000 complete [ 3323.874171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.881864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3323.897388] Lustre: Skipped 1 previous similar message [ 3323.912642] Lustre: Skipped 2 previous similar messages [ 3329.001276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.010909] Lustre: Skipped 1 previous similar message [ 3329.854749] Lustre: server umount lustre-MDT0000 complete [ 3333.091784] LustreError: 118368:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.114465] LustreError: 118368:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3333.905819] LustreError: 115855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857422 with bad export cookie 12507117444494847743 [ 3333.926331] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3333.929354] LustreError: 115855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3334.422944] Lustre: server umount lustre-MDT0001 complete [ 3345.450588] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 3347.918903] Key type lgssc unregistered [ 3348.281866] LNet: 119410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3348.286949] LNetError: 119410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3348.300205] LNet: Removed LNI 192.168.206.129@tcp [ 3349.105974] Key type .llcrypt unregistered [ 3349.115496] Key type ._llcrypt unregistered [ 3371.388236] Key type ._llcrypt registered [ 3371.392287] Key type .llcrypt registered [ 3371.514546] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 3372.720987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3372.798945] alg: No test for adler32 (adler32-zlib) [ 3373.912625] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3374.201488] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 3375.864165] Key type lgssc registered [ 3377.057891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3384.346987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3384.376020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.745595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.108614] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3396.065832] LustreError: 120831:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.088392] LustreError: 120831:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3398.307326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.559175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3402.286356] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3409.697546] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3415.080707] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3421.432806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.766987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.730993] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3430.961191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3435.099819] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3443.296686] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3447.618457] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3447.632323] Lustre: Skipped 2 previous similar messages [ 3452.085334] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3452.098239] Lustre: Skipped 2 previous similar messages [ 3452.547335] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3452.631990] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3458.376735] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.29@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3458.388086] Lustre: Skipped 2 previous similar messages [ 3460.208429] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.29@tcp 192.168.0.[2,10]@tcp [ 3465.435498] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.29/24@tcp in log lustre-MDT0000 [ 3465.442447] Lustre: Skipped 2 previous similar messages [ 3465.655262] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.29/24@tcp [ 3465.670653] Lustre: Skipped 1 previous similar message [ 3481.075556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3481.082371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.088753] LustreError: Skipped 1 previous similar message [ 3481.135558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.091056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.107201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.132404] Lustre: Skipped 1 previous similar message [ 3482.429962] Lustre: server umount lustre-OST0000 complete [ 3487.201868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.207943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.219574] Lustre: Skipped 1 previous similar message [ 3487.238330] Lustre: Skipped 1 previous similar message [ 3492.322767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3492.328788] Lustre: Skipped 1 previous similar message [ 3492.695945] Lustre: server umount lustre-MDT0000 complete [ 3497.448196] LustreError: 120827:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.470611] LustreError: 120827:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3497.950574] LustreError: 120813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857586 with bad export cookie 17981523107858918184 [ 3497.957352] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.967653] LustreError: 120813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3498.394346] Lustre: server umount lustre-MDT0001 complete [ 3511.452916] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 3515.235378] Key type lgssc unregistered [ 3515.612603] LNet: 125452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3515.620962] LNetError: 125452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3515.633899] LNet: Removed LNI 192.168.206.129@tcp [ 3516.417022] Key type .llcrypt unregistered [ 3516.418688] Key type ._llcrypt unregistered [ 3546.696422] Key type ._llcrypt registered [ 3546.698400] Key type .llcrypt registered [ 3546.826671] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 3547.806938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3547.858073] alg: No test for adler32 (adler32-zlib) [ 3548.917776] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3549.195074] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 3550.848166] Key type lgssc registered [ 3551.709258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3559.488705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3559.527028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.006844] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3561.039682] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.29/24@tcp [ 3561.054905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.365198] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3570.150378] LustreError: 127012:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.171482] LustreError: 127012:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3573.832144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.246363] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3574.300324] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.29/24@tcp [ 3574.358624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3578.624714] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3586.421332] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3591.998417] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3599.378668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.796708] LustreError: 128840:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.086814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.237339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3606.900146] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3614.197962] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3625.955828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3625.976856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.994534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.000869] Lustre: Skipped 1 previous similar message [ 3629.561184] Lustre: server umount lustre-OST0000 complete [ 3635.681772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3635.689750] LustreError: Skipped 1 previous similar message [ 3635.694305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.709546] Lustre: Skipped 1 previous similar message [ 3635.718207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.329905] Lustre: server umount lustre-MDT0000 complete [ 3641.316411] LustreError: 127013:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.328781] LustreError: 127013:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3643.318434] LustreError: 126993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857732 with bad export cookie 15193279848174997573 [ 3643.325557] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3643.331112] LustreError: 126993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3643.730481] Lustre: server umount lustre-MDT0001 complete [ 3654.643865] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 3657.509572] Key type lgssc unregistered [ 3657.763500] LNet: 130603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3657.769115] LNetError: 130603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3657.790595] LNet: Removed LNI 192.168.206.129@tcp [ 3658.638662] Key type .llcrypt unregistered [ 3658.641858] Key type ._llcrypt unregistered [ 3683.541300] Key type ._llcrypt registered [ 3683.544247] Key type .llcrypt registered [ 3683.634368] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 3697.985440] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 3698.719776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3698.773525] alg: No test for adler32 (adler32-zlib) [ 3699.803893] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 3699.995530] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 3701.704179] Key type lgssc registered [ 3703.024682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3736.302382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3736.344410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.626723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3737.661525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3737.730287] Lustre: lustre-MDT0000: new disk, initializing [ 3737.810993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.826243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3741.803864] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3752.618529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.759256] Lustre: 134004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3752.820968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3752.824638] Lustre: Skipped 1 previous similar message [ 3752.880306] Lustre: lustre-MDT0001: new disk, initializing [ 3752.960355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3753.004488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3753.018404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3757.350784] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3765.583407] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3769.873877] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3775.392601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3775.623875] Lustre: lustre-OST0000: new disk, initializing [ 3775.627556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3775.632713] Lustre: 135972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3775.707151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.013250] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3781.700777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3781.717904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3781.839713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3789.671360] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3794.023471] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3794.252310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3798.875329] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3799.081747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3802.595976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.599805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.607548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.626612] Lustre: Skipped 1 previous similar message [ 3807.724749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.736041] Lustre: Skipped 1 previous similar message [ 3812.834797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.853411] Lustre: Skipped 2 previous similar messages [ 3814.880543] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3814.978989] Lustre: server umount lustre-OST0000 complete [ 3819.493594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3819.503899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.518764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.073305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.785032] Lustre: server umount lustre-MDT0000 complete [ 3828.251031] LustreError: 135971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857917 with bad export cookie 7011577896878803834 [ 3828.265760] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.273521] LustreError: 135971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3828.605658] Lustre: server umount lustre-MDT0001 complete [ 3837.110862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.528386] LustreError: 138189:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.540691] LustreError: 138189:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 3837.596539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.631341] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3843.056228] LustreError: 138190:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.139165] LustreError: 138189:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.057686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.293291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.584614] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3861.494232] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3866.991432] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3874.437975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.873259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.819278] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3890.185945] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3895.207839] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3895.426317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3900.184816] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3900.431100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3910.506840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3910.715621] Lustre: lustre-OST0001: new disk, initializing [ 3910.721519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3910.730884] Lustre: 141560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3910.847848] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3917.835498] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 3918.453223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3918.467553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3918.574161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3927.240522] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3927.443507] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3932.098297] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3932.348442] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3938.120311] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ff245b03000.ost_server_uuid 50 [ 3939.573812] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ff245b03000.ost_server_uuid in FULL state after 0 sec [ 3954.684306] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3954.684395] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.697978] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3954.715188] Lustre: Skipped 1 previous similar message [ 3954.741372] Lustre: Skipped 3 previous similar messages [ 3955.839107] Lustre: server umount lustre-OST0001 complete [ 3959.781840] LustreError: 140376:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.830296] LustreError: 140376:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3961.828147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3961.842197] LustreError: Skipped 1 previous similar message [ 3961.852413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.900383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.904636] LustreError: 140009:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.914808] Lustre: Skipped 2 previous similar messages [ 3964.939917] LustreError: 140009:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3966.896902] Lustre: server umount lustre-OST0000 complete [ 3972.066091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3972.073341] LustreError: Skipped 1 previous similar message [ 3972.077671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.095346] Lustre: Skipped 1 previous similar message [ 3976.949243] Lustre: server umount lustre-MDT0000 complete [ 3980.273257] LustreError: 138185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.289902] LustreError: 138185:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3980.771337] LustreError: 141570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858069 with bad export cookie 7011577896878804716 [ 3980.794705] LustreError: 141570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3980.797494] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3981.046225] Lustre: server umount lustre-MDT0001 complete [ 3999.449560] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 4001.914915] Key type lgssc unregistered [ 4002.254187] LNet: 144566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4002.268259] LNetError: 144566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4002.286341] LNet: Removed LNI 192.168.206.129@tcp [ 4003.282981] Key type .llcrypt unregistered [ 4003.285283] Key type ._llcrypt unregistered [ 4037.947273] Key type ._llcrypt registered [ 4037.949627] Key type .llcrypt registered [ 4038.056732] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 4051.551308] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 4052.529276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4052.628868] alg: No test for adler32 (adler32-zlib) [ 4053.729470] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4054.097377] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 4055.752223] Key type lgssc registered [ 4056.704281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4090.900909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4090.924317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.220202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4092.253656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4092.343664] Lustre: lustre-MDT0000: new disk, initializing [ 4092.418210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.431148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4096.759303] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4106.578473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.702400] Lustre: 148720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4106.772462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4106.778622] Lustre: Skipped 1 previous similar message [ 4106.884714] Lustre: lustre-MDT0001: new disk, initializing [ 4107.007335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4107.068850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4107.080248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4111.239585] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4120.215808] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4125.009252] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4131.693631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.955525] Lustre: lustre-OST0000: new disk, initializing [ 4131.961909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4131.966705] Lustre: 150689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4132.046380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.473558] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4140.627452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4140.645303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4140.728335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4148.366705] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4152.355973] Lustre: Failing over lustre-OST0000 [ 4154.608445] Lustre: server umount lustre-OST0000 complete [ 4170.721099] Lustre: 146942:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858243/real 1787858243] req@0000000044e9ab3b x1874705140188160/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787858259 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4170.756321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.422365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4172.713315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.753251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.779646] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4173.728092] Lustre: 146939:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858245/real 1787858245] req@00000000c9f8c8dc x1874705140188544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787858261 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4175.781592] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4177.825084] Lustre: 146939:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858250/real 1787858250] req@00000000e610786e x1874705140189952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787858266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4177.871811] Lustre: 146939:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4178.877752] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4181.009192] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4181.011475] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4182.848215] Lustre: Failing over lustre-MDT0000 [ 4183.077141] Lustre: server umount lustre-MDT0000 complete [ 4183.905159] Lustre: 146939:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858255/real 1787858255] req@00000000cfd45a85 x1874705140190976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787858271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4184.053773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.072120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.087374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.106549] Lustre: Skipped 1 previous similar message [ 4191.151660] LustreError: 148728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.183752] LustreError: 148728:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 4196.274773] LustreError: 148727:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.308245] LustreError: 148727:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4201.413618] LustreError: 149714:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.445223] Lustre: 146939:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858274/real 1787858275] req@00000000f85b7306 x1874705140201472/t0(0) o400->MGC192.168.206.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787858290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4201.495282] Lustre: 146939:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4201.518616] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.967091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.317205] LustreError: 149714:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.337709] LustreError: 149714:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4211.682327] LustreError: 149715:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.689785] LustreError: 146938:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007ca76b7a x1874705140206592/t0(0) o250->MGC192.168.206.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4211.720045] LustreError: 149715:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 4212.062425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.133890] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4212.461681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4216.689164] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4217.338840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4217.349536] Lustre: Skipped 1 previous similar message [ 4217.366325] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4217.411415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4227.045438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4227.050896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.060693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4227.563401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.676090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.683588] Lustre: Skipped 1 previous similar message [ 4237.794509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.304139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4238.424652] Lustre: server umount lustre-OST0000 complete [ 4242.920982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.929262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4242.947980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4242.949854] Lustre: Skipped 2 previous similar messages [ 4242.979675] Lustre: Skipped 2 previous similar messages [ 4253.153229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.158452] Lustre: Skipped 3 previous similar messages [ 4256.224187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4256.418133] Lustre: server umount lustre-MDT0000 complete [ 4258.274077] LustreError: 148732:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.286994] LustreError: 148732:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4260.095362] LustreError: 148713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858349 with bad export cookie 5496922619386631532 [ 4260.101169] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.120451] LustreError: 148713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4260.517627] Lustre: server umount lustre-MDT0001 complete [ 4271.261716] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 4273.855168] Key type lgssc unregistered [ 4274.095655] LNet: 154348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4274.105455] LNetError: 154348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4274.116165] LNet: Removed LNI 192.168.206.129@tcp [ 4274.904358] Key type .llcrypt unregistered [ 4274.905912] Key type ._llcrypt unregistered [ 4309.952906] Key type ._llcrypt registered [ 4309.955078] Key type .llcrypt registered [ 4310.050417] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 4311.059689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4311.085322] alg: No test for adler32 (adler32-zlib) [ 4312.169856] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 4312.421605] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 4314.096220] Key type lgssc registered [ 4315.050627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4322.347186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4322.397234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.972825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.856931] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4333.538311] LustreError: 155869:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4333.577065] LustreError: 155869:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4336.258551] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.669741] LustreError: 155868:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.474300] LustreError: 155898:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4340.491624] LustreError: 155898:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4340.499394] LustreError: 155898:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4340.907554] Lustre: server umount lustre-MDT0000 complete [ 4358.484723] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 4371.827163] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 4412.562355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.880768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4412.914108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4413.043090] Lustre: lustre-MDT0000: new disk, initializing [ 4413.242725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.282104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4418.433282] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4430.162921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.357669] Lustre: 159203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4430.460740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4430.467700] Lustre: Skipped 1 previous similar message [ 4430.648372] Lustre: lustre-MDT0001: new disk, initializing [ 4430.788972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4430.842567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4430.869939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4436.379137] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4445.459791] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4450.528620] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4456.677329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4456.882309] Lustre: lustre-OST0000: new disk, initializing [ 4456.885623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4456.890677] Lustre: 161171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4456.966278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.411547] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4466.814091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4466.828331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4466.947761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4471.348322] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4475.150698] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4477.216287] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:22:43 (1787858563) [ 4691.936807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4691.944064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.993663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.450294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.484426] Lustre: Skipped 1 previous similar message [ 4692.505316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.525202] Lustre: Skipped 2 previous similar messages [ 4696.355586] Lustre: server umount lustre-MDT0000 complete [ 4697.569756] LustreError: 159215:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.612218] LustreError: 159215:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4700.588940] LustreError: 159196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858789 with bad export cookie 17806769496066986741 [ 4700.591657] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4700.605549] LustreError: 159196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4702.691430] LustreError: 159212:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.705271] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.743950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4705.701201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4707.043532] Lustre: server umount lustre-MDT0001 complete [ 4717.739485] Lustre: server umount lustre-OST0000 complete [ 4727.296834] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 4738.900263] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 4774.482718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.717520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4774.757791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4774.823945] Lustre: lustre-MDT0000: new disk, initializing [ 4774.892899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.910358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4779.229171] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4789.031988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.151879] Lustre: 165825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4789.219390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4789.222398] Lustre: Skipped 1 previous similar message [ 4789.301281] Lustre: lustre-MDT0001: new disk, initializing [ 4789.428557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4789.470099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4789.479982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4793.930790] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4802.120400] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4807.017710] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4813.793892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.071765] Lustre: lustre-OST0000: new disk, initializing [ 4814.079073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4814.091254] Lustre: 167795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4814.163921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.605871] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4821.071305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4821.083980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4821.261155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4829.019845] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4834.983731] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4835.361487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4840.092628] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4840.315067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4841.962851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4841.972687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.978220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4841.986764] Lustre: Skipped 1 previous similar message [ 4841.999061] Lustre: Skipped 1 previous similar message [ 4852.196895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4852.207609] Lustre: Skipped 3 previous similar messages [ 4856.290514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4856.540858] Lustre: server umount lustre-OST0000 complete [ 4861.410298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4861.419931] LustreError: Skipped 1 previous similar message [ 4861.431949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.809913] Lustre: server umount lustre-MDT0000 complete [ 4867.553553] LustreError: 165837:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.578546] LustreError: 165837:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4870.390850] LustreError: 165819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858959 with bad export cookie 17806769496067145571 [ 4870.392882] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4870.398758] LustreError: 165819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4870.689230] Lustre: server umount lustre-MDT0001 complete [ 4878.216440] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:29:25 (1787858965) [ 4893.180562] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 4901.512791] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 4933.314760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.547208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4933.583172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4933.642440] Lustre: lustre-MDT0000: new disk, initializing [ 4933.700063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.715443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4937.663801] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4948.658043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.786108] Lustre: 172551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4949.009973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4953.481864] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4963.070544] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4969.578912] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4976.202839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4976.522917] Lustre: lustre-OST0000: new disk, initializing [ 4976.526594] Lustre: Skipped 1 previous similar message [ 4976.530764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4976.535066] Lustre: Skipped 2 previous similar messages [ 4976.539523] Lustre: 174526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4982.872504] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 4985.927704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4985.940484] Lustre: Skipped 1 previous similar message [ 4985.946203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4986.096411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4992.966454] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5001.698107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5001.705566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.710510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5001.721718] Lustre: Skipped 2 previous similar messages [ 5001.741300] Lustre: Skipped 4 previous similar messages [ 5005.786791] Lustre: server umount lustre-OST0000 complete [ 5010.401203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5015.417023] Lustre: server umount lustre-MDT0000 complete [ 5017.057613] LustreError: 172562:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.078895] LustreError: 172562:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5018.656613] LustreError: 172544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859107 with bad export cookie 17806769496067146453 [ 5018.658289] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.668737] LustreError: 172544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5019.044179] Lustre: server umount lustre-MDT0001 complete [ 5026.865166] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:31:53 (1787859113) [ 5041.843513] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 5051.048836] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 5087.232025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.532754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5087.572862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5087.662788] Lustre: lustre-MDT0000: new disk, initializing [ 5087.754851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.761975] Lustre: Skipped 2 previous similar messages [ 5087.778593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5092.621469] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5102.989296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.138728] Lustre: 178849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5103.425594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5108.182270] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5118.005510] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5124.778456] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5132.583315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5133.035669] Lustre: 180827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5142.302103] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5142.641522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5142.827876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5154.331563] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5168.610952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5168.621869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.628921] LustreError: Skipped 1 previous similar message [ 5168.643056] Lustre: Skipped 2 previous similar messages [ 5168.648751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5168.648758] Lustre: Skipped 3 previous similar messages [ 5169.885623] Lustre: server umount lustre-OST0000 complete [ 5179.662829] Lustre: server umount lustre-MDT0000 complete [ 5183.255364] LustreError: 178840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859272 with bad export cookie 17806769496067147538 [ 5183.291372] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5183.869774] Lustre: server umount lustre-MDT0001 complete [ 5195.347852] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 5198.434404] Key type lgssc unregistered [ 5198.872599] LNet: 182687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5198.882208] LNetError: 182687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5198.905415] LNet: Removed LNI 192.168.206.129@tcp [ 5199.729595] Key type .llcrypt unregistered [ 5199.735946] Key type ._llcrypt unregistered [ 5224.431700] Key type ._llcrypt registered [ 5224.434206] Key type .llcrypt registered [ 5224.570924] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 5225.712461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5225.758213] alg: No test for adler32 (adler32-zlib) [ 5226.896345] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5227.272210] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 5228.985725] Key type lgssc registered [ 5230.493695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5238.079073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5238.109668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.528504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.410407] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5250.019297] LustreError: 184106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5250.042441] LustreError: 184106:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5252.138060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.653315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5257.180733] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5264.010757] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5269.329861] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5275.286327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.673285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.836742] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5284.915553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5288.790391] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5310.442386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.445776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5310.467417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5310.488434] Lustre: Skipped 1 previous similar message [ 5315.552803] LustreError: 185929:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.954081] Lustre: server umount lustre-OST0000 complete [ 5320.675450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.678420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.692024] Lustre: Skipped 2 previous similar messages [ 5320.703296] Lustre: Skipped 1 previous similar message [ 5325.794263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5325.801644] Lustre: Skipped 1 previous similar message [ 5326.156198] Lustre: server umount lustre-MDT0000 complete [ 5329.957483] LustreError: 184087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859418 with bad export cookie 17898127001068663823 [ 5329.965987] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.229376] Lustre: server umount lustre-MDT0001 complete [ 5343.159828] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 5347.101236] Key type lgssc unregistered [ 5347.496206] LNet: 187664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5347.506524] LNetError: 187664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5347.529445] LNet: Removed LNI 192.168.206.129@tcp [ 5348.398198] Key type .llcrypt unregistered [ 5348.400426] Key type ._llcrypt unregistered [ 5372.672609] Key type ._llcrypt registered [ 5372.675336] Key type .llcrypt registered [ 5372.790986] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 5373.936237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5374.001942] alg: No test for adler32 (adler32-zlib) [ 5375.232398] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 5375.572862] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 5377.280156] Key type lgssc registered [ 5378.699591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5385.977749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5386.024044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.602926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.183422] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5397.985183] LustreError: 189083:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.000788] LustreError: 189083:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5399.612914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.875504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.026902] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5410.192687] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5415.298192] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5421.813608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5422.170183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.337563] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5432.431782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5437.472764] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.103791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.117834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.138717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5448.171374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5448.172709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5448.187666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.189765] Lustre: Skipped 2 previous similar messages [ 5450.668674] Lustre: lustre-OST0000: Not available for connect from 192.168.206.29@tcp (stopping) [ 5450.817630] Lustre: server umount lustre-OST0000 complete [ 5457.349323] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ff247fe6800.ost_server_uuid 50 [ 5458.859144] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5473.585695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.596532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5473.624876] Lustre: Skipped 1 previous similar message [ 5478.882313] LustreError: 191334:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.902946] LustreError: 191334:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5480.867884] Lustre: server umount lustre-MDT0000 complete [ 5484.001745] LustreError: 189079:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.022109] LustreError: 189079:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5484.871756] LustreError: 189063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859573 with bad export cookie 7683551107189705245 [ 5484.884276] LustreError: 189063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5484.885782] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5485.183841] Lustre: server umount lustre-MDT0001 complete [ 5494.181896] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:39:40 (1787859580) [ 5502.436722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.445366] LustreError: 192998:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.616331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.579080] LustreError: 192999:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.867587] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5518.840991] LustreError: 192998:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.887439] LustreError: 192998:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5521.055046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.547784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5526.660538] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5535.008780] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5541.461675] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5547.811835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.208679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.239576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5553.977588] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5560.555796] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5567.873791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5568.035112] Lustre: lustre-OST0001: new disk, initializing [ 5568.039992] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5568.046348] Lustre: 195840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5568.131171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5573.411312] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5577.316533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5577.325207] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5577.395817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5581.791611] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5585.378798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5585.386125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.404246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5585.414886] Lustre: Skipped 1 previous similar message [ 5587.427139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5590.517632] Lustre: server umount lustre-OST0000 complete [ 5592.490706] LustreError: 194822:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.768395] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5595.965821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5600.462684] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5600.697524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5618.657123] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5618.673059] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.674715] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5618.685641] Lustre: Skipped 2 previous similar messages [ 5618.696225] Lustre: Skipped 3 previous similar messages [ 5620.625067] Lustre: server umount lustre-OST0001 complete [ 5638.112134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5638.114884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5638.130186] Lustre: Skipped 8 previous similar messages [ 5638.308345] Lustre: server umount lustre-MDT0000 complete [ 5641.425734] LustreError: 192979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859730 with bad export cookie 7683551107189706316 [ 5641.431135] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5641.433799] LustreError: 192979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5641.696287] Lustre: server umount lustre-MDT0001 complete [ 5661.799054] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:42:28 (1787859748) [ 5668.595807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.901489] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5668.938621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5669.124248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.845155] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5680.721805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.792723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5680.814839] Lustre: 199279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5684.765534] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5692.111423] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5698.276905] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.015565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.216242] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5712.181365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5713.444208] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5720.819645] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5728.590904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5728.659633] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5734.410447] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5734.585732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5741.671340] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5745.135440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.137448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.156856] Lustre: Skipped 3 previous similar messages [ 5745.166680] Lustre: Skipped 1 previous similar message [ 5750.400741] Lustre: server umount lustre-OST0000 complete [ 5752.754603] LustreError: 201131:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.779215] LustreError: 201131:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 15 previous similar messages [ 5770.733798] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5770.735434] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.739835] LustreError: Skipped 2 previous similar messages [ 5770.756513] Lustre: Skipped 1 previous similar message [ 5772.282901] Lustre: server umount lustre-OST0001 complete [ 5790.182946] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5790.350852] Lustre: server umount lustre-MDT0000 complete [ 5794.161278] LustreError: 201150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859883 with bad export cookie 7683551107189707723 [ 5794.170363] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.174260] LustreError: 201150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5794.644174] Lustre: server umount lustre-MDT0001 complete [ 5819.967296] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:45:06 (1787859906) [ 5836.121851] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 5845.435163] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 5880.923568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.153231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5881.180035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5881.259574] Lustre: lustre-MDT0000: new disk, initializing [ 5881.329545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.338083] Lustre: Skipped 3 previous similar messages [ 5881.361336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5884.901751] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5895.078959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.184930] Lustre: 207377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5895.231497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5895.234888] Lustre: Skipped 1 previous similar message [ 5895.316941] Lustre: lustre-MDT0001: new disk, initializing [ 5895.469505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5895.485096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5900.947315] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5910.303859] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5916.267046] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5922.522926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.862734] Lustre: lustre-OST0000: new disk, initializing [ 5922.868730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5922.876651] Lustre: 209345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5929.292419] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5932.614207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5932.636695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5932.819161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5938.203884] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5942.593165] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5942.757737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5946.791564] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5947.030628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5953.506295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.511082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5953.515412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5953.515420] Lustre: Skipped 13 previous similar messages [ 5953.522774] Lustre: Skipped 3 previous similar messages [ 5953.550161] LustreError: Skipped 1 previous similar message [ 5963.232398] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5963.417151] Lustre: server umount lustre-OST0000 complete [ 5973.364465] Lustre: server umount lustre-MDT0000 complete [ 5973.991216] LustreError: 207388:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.009114] LustreError: 207388:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 5977.071039] LustreError: 207370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860065 with bad export cookie 7683551107189709508 [ 5977.082503] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5977.088528] LustreError: 207370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5977.673464] Lustre: server umount lustre-MDT0001 complete [ 5985.820310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.393038] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 5999.504761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.649979] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6012.398482] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6018.657445] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6024.900095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.197523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6025.205825] Lustre: Skipped 4 previous similar messages [ 6030.613478] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6038.201104] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6042.799323] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6043.039496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6048.101784] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6048.367315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6051.809864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6051.814712] LustreError: Skipped 1 previous similar message [ 6051.820602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.837923] Lustre: Skipped 2 previous similar messages [ 6056.438978] Lustre: server umount lustre-OST0000 complete [ 6061.867451] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6065.723829] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6094.806094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6101.705729] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6104.423707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.444521] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6108.596670] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6113.149237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6117.047862] Lustre: server umount lustre-OST0000 complete [ 6122.980218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6122.994561] LustreError: Skipped 1 previous similar message [ 6123.005542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.011248] Lustre: Skipped 9 previous similar messages [ 6130.273978] Lustre: server umount lustre-MDT0000 complete [ 6134.242830] LustreError: 211564:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.261076] LustreError: 211564:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 6134.761307] LustreError: 211545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860223 with bad export cookie 7683551107189710390 [ 6134.763045] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.770192] LustreError: 211545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6135.203829] Lustre: server umount lustre-MDT0001 complete [ 6144.495518] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:50:31 (1787860231) [ 6150.563319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.590980] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6163.446923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.469422] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6176.064556] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6183.165797] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6188.768663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.895261] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6202.369214] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.622624] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6206.797137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6211.285536] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6211.503730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6217.344232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.502420] Lustre: lustre-OST0001: new disk, initializing [ 6217.508962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6217.516438] Lustre: 220904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6223.450913] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6223.503631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6223.516552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6223.654112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6232.292319] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6236.620818] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6236.848466] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6241.162968] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6241.435757] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6244.321633] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.336055] Lustre: Skipped 4 previous similar messages [ 6249.476765] Lustre: server umount lustre-OST0001 complete [ 6255.382976] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6255.622187] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6259.725519] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6259.949087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6288.672026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.880472] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6288.899313] Lustre: Skipped 5 previous similar messages [ 6293.999809] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6294.028277] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6294.028810] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6294.059313] Lustre: Skipped 1 previous similar message [ 6295.269944] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6302.248667] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6303.978561] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6309.347021] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6309.357803] LustreError: Skipped 2 previous similar messages [ 6320.096150] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6320.283145] Lustre: server umount lustre-OST0001 complete [ 6351.329150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6351.514489] Lustre: server umount lustre-OST0000 complete [ 6361.546494] Lustre: server umount lustre-MDT0000 complete [ 6365.308610] LustreError: 220903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860454 with bad export cookie 7683551107189711349 [ 6365.316937] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6365.319652] LustreError: 220903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6371.720924] Lustre: server umount lustre-MDT0001 complete [ 6393.324612] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:54:40 (1787860480) [ 6399.544266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.984155] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6400.024719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.492417] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6413.910311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6414.049827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6414.078503] Lustre: 226149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6418.704963] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6426.623938] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6433.364801] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6440.213390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.464236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6446.498935] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6453.573398] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6462.508987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.622537] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6468.649960] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6474.842649] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.082119] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6480.992275] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6481.182307] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6485.472946] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6485.685739] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6491.044634] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ff247fe5800.ost_server_uuid 50 [ 6492.449635] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ff247fe5800.ost_server_uuid in IDLE state after 0 sec [ 6493.436873] Lustre: Permanently deactivating lustre-OST0001 [ 6493.456825] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6497.193318] Lustre: Permanently reactivating lustre-OST0001 [ 6503.419214] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.439237] Lustre: Skipped 6 previous similar messages [ 6503.458536] LustreError: 227994:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.481735] LustreError: 227994:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 39 previous similar messages [ 6503.594714] Lustre: server umount lustre-OST0001 complete [ 6507.492318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6507.500993] Lustre: Skipped 26 previous similar messages [ 6513.551552] Lustre: server umount lustre-OST0000 complete [ 6523.190610] Lustre: server umount lustre-MDT0000 complete [ 6526.098568] LustreError: 228009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860615 with bad export cookie 7683551107189712637 [ 6526.108151] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6526.119887] LustreError: 228009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6526.417342] Lustre: server umount lustre-MDT0001 complete [ 6547.292397] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:57:14 (1787860634) [ 6553.992494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.299423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6554.332323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6554.340339] Lustre: Skipped 5 previous similar messages [ 6558.619429] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6566.963551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.080805] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6567.129490] Lustre: 232838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6571.583203] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6579.358919] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6586.297980] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6594.949238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.299667] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6595.350836] Lustre: Permanently deactivating lustre-OST0000 [ 6595.365412] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6600.775391] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6600.791568] LustreError: 234984:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 6602.365814] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6609.721476] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.830509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6615.910741] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6621.885444] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6625.492966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6628.476305] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.382395] Lustre: Permanently reactivating lustre-OST0000 [ 6630.390686] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6630.397160] Lustre: Skipped 2 previous similar messages [ 6635.118047] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6635.119693] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000d0c95782, cur 1787860724 deadline 1787860719 last 1787860689 [ 6635.130344] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6635.164920] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6635.243789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6651.362118] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.377292] LustreError: Skipped 5 previous similar messages [ 6653.144713] Lustre: server umount lustre-OST0001 complete [ 6662.673328] Lustre: server umount lustre-OST0000 complete [ 6672.606439] Lustre: server umount lustre-MDT0000 complete [ 6676.268733] LustreError: 232830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860765 with bad export cookie 7683551107189714289 [ 6676.274741] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6676.290195] LustreError: 232830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6676.683493] Lustre: server umount lustre-MDT0001 complete [ 6686.333609] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 6689.076644] Key type lgssc unregistered [ 6689.404913] LNet: 237721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6689.410439] LNetError: 237721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6689.433423] LNet: Removed LNI 192.168.206.129@tcp [ 6690.181483] Key type .llcrypt unregistered [ 6690.191994] Key type ._llcrypt unregistered [ 6712.851732] Key type ._llcrypt registered [ 6712.856118] Key type .llcrypt registered [ 6712.955484] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 6714.188040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6714.336028] alg: No test for adler32 (adler32-zlib) [ 6715.557399] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 6715.894294] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 6717.656196] Key type lgssc registered [ 6718.878413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6729.774321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6729.807718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.216083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6735.352553] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6741.483822] LustreError: 239188:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.537924] LustreError: 239188:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6746.300653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.455396] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6746.480329] Lustre: 239179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6746.498206] Lustre: Permanently deactivating lustre-MDT0001 [ 6746.513248] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6746.594655] LustreError: 239189:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.925513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6752.219491] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6754.835300] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6760.627341] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6767.139921] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6774.607481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6774.875628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.938622] LustreError: 241042:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.999337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6782.436383] LustreError: 241042:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.801743] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6787.567767] LustreError: 241040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.698098] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6791.841714] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.774747] LustreError: 241151:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6796.803589] LustreError: 241151:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6799.302043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6799.641309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6805.009042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6805.017069] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6806.517391] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6814.591725] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6817.480859] Lustre: Permanently reactivating lustre-MDT0001 [ 6817.502322] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6817.510733] Lustre: Skipped 2 previous similar messages [ 6824.411552] Lustre: lustre-MDT0001: Client f938c65b-c223-4e46-b859-4f8c25f87620 (at 192.168.206.29@tcp) reconnecting [ 6826.533850] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6826.540885] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6826.544288] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6826.549213] LustreError: 242814:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6826.553538] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6826.554849] LustreError: 242814:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6826.554968] LustreError: 242814:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.129@tcp: cfg command failed: rc = -108 [ 6826.554988] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6826.554988] [ 6826.562339] LustreError: 239182:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6830.606042] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6832.368581] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6837.954197] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6838.180387] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6842.497221] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6842.759274] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6845.684956] Lustre: Permanently deactivating lustre-MDT0001 [ 6845.705593] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6845.714099] Lustre: Skipped 2 previous similar messages [ 6853.200418] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6864.866970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6864.877145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.898246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.435806] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.440956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.455705] Lustre: Skipped 1 previous similar message [ 6867.470342] Lustre: Skipped 3 previous similar messages [ 6869.020881] Lustre: server umount lustre-MDT0000 complete [ 6872.375777] LustreError: 239169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860961 with bad export cookie 12059292196671345924 [ 6872.377121] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6872.383955] LustreError: 239169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6872.546469] LustreError: 239184:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.589361] LustreError: 239184:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6872.714394] Lustre: server umount lustre-MDT0001 complete [ 6882.303750] Lustre: server umount lustre-OST0000 complete [ 6892.042619] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:02:59 (1787860979) [ 6901.510742] Lustre: server umount lustre-OST0001 complete [ 6907.243861] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 6916.100153] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 6948.175965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.389471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6948.405856] Lustre: Skipped 2 previous similar messages [ 6948.433357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6948.505852] Lustre: lustre-MDT0000: new disk, initializing [ 6948.588394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.608801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6952.572871] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6962.514565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.672208] Lustre: 247477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6962.762645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6962.772441] Lustre: Skipped 1 previous similar message [ 6962.877346] Lustre: lustre-MDT0001: new disk, initializing [ 6963.003204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6963.035983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6963.050050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6967.809387] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6976.619615] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6981.615737] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6988.380438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.582742] Lustre: lustre-OST0000: new disk, initializing [ 6988.589193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6988.597541] Lustre: 249445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6988.688682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.996844] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 6998.627855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6998.640463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6998.767104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7004.586836] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7007.042069] LustreError: 247483:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7016.553291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.709843] Lustre: lustre-OST0001: new disk, initializing [ 7016.713970] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7016.721121] Lustre: 250559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7017.084809] LustreError: 247483:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7017.126573] LustreError: 247484:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7022.340204] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7023.667839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7023.680347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7023.799284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7027.216125] LustreError: 247484:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7031.647910] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.210472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7036.883423] Lustre: lustre-OST0001: Not available for connect from 192.168.206.29@tcp (stopping) [ 7039.462929] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7039.463348] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.471017] LustreError: Skipped 1 previous similar message [ 7039.474961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7039.483549] Lustre: Skipped 2 previous similar messages [ 7039.497620] Lustre: Skipped 1 previous similar message [ 7041.200344] Lustre: server umount lustre-OST0001 complete [ 7041.974350] LustreError: 249430:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.940557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.972054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7059.981745] Lustre: Skipped 1 previous similar message [ 7063.542595] Lustre: server umount lustre-OST0000 complete [ 7070.177977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.193410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.196542] Lustre: Skipped 2 previous similar messages [ 7070.218449] Lustre: Skipped 1 previous similar message [ 7073.568561] Lustre: server umount lustre-MDT0000 complete [ 7077.307117] LustreError: 247469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861166 with bad export cookie 12059292196671348738 [ 7077.323866] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.327342] LustreError: 247469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7083.728414] Lustre: server umount lustre-MDT0001 complete [ 7094.345135] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 7097.061825] Key type lgssc unregistered [ 7097.368628] LNet: 252595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7097.372458] LNetError: 252595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7097.395879] LNet: Removed LNI 192.168.206.129@tcp [ 7098.276701] Key type .llcrypt unregistered [ 7098.287526] Key type ._llcrypt unregistered [ 7137.494217] Key type ._llcrypt registered [ 7137.498241] Key type .llcrypt registered [ 7137.616236] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 7138.961977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7139.139643] alg: No test for adler32 (adler32-zlib) [ 7140.275778] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7140.597662] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 7142.280165] Key type lgssc registered [ 7143.484068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7152.167469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7152.202371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.586439] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7153.626746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7153.870534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7158.372925] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7167.769399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.872222] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7167.926291] Lustre: 254806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7168.180294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7172.765700] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7179.605798] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.497705] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7190.972596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.188830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7191.358636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.163403] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7200.524338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7204.508826] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7257.061117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7257.068088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.078942] LustreError: Skipped 1 previous similar message [ 7257.090353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7257.123428] Lustre: Skipped 1 previous similar message [ 7262.000508] Lustre: server umount lustre-OST0000 complete [ 7267.324380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7280.410351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7280.715503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.353450] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7294.621800] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7298.082882] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7298.095928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7298.108459] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7298.123530] Lustre: Skipped 1 previous similar message [ 7333.863898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7333.868689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.871598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7333.881530] Lustre: Skipped 1 previous similar message [ 7333.890840] Lustre: Skipped 2 previous similar messages [ 7338.981600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7338.995379] Lustre: Skipped 1 previous similar message [ 7340.001517] Lustre: server umount lustre-OST0000 complete [ 7344.098271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.111700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7344.114166] Lustre: Skipped 1 previous similar message [ 7349.220224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7349.228167] Lustre: Skipped 1 previous similar message [ 7350.206243] Lustre: server umount lustre-MDT0000 complete [ 7354.003875] LustreError: 255548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861442 with bad export cookie 9178174034684060226 [ 7354.019329] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7354.034632] LustreError: 255548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7354.414042] Lustre: server umount lustre-MDT0001 complete [ 7366.763421] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 7370.216951] Key type lgssc unregistered [ 7370.516741] LNet: 260239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7370.530746] LNetError: 260239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7370.563080] LNet: Removed LNI 192.168.206.129@tcp [ 7371.637913] Key type .llcrypt unregistered [ 7371.640450] Key type ._llcrypt unregistered [ 7397.158931] Key type ._llcrypt registered [ 7397.164956] Key type .llcrypt registered [ 7397.248819] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 7397.989697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7398.004937] alg: No test for adler32 (adler32-zlib) [ 7399.235190] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7399.473764] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 7401.160142] Key type lgssc registered [ 7402.292233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7410.383339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7410.414898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.923445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.338184] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7421.416353] LustreError: 261654:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.434078] LustreError: 261654:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7423.879250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7424.326100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7428.674403] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7435.012990] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7439.988397] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7446.057816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.416277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.829670] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7456.712625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7459.958655] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7485.410405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7485.416537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.436460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7487.467970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7487.472869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7487.481166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.324413] Lustre: server umount lustre-OST0000 complete [ 7496.162733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7496.172877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.203155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.209383] Lustre: Skipped 1 previous similar message [ 7499.141312] Lustre: server umount lustre-MDT0000 complete [ 7502.825664] LustreError: 264144:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.840522] LustreError: 264144:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7502.923044] LustreError: 261634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861591 with bad export cookie 7365607755287464159 [ 7502.935973] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7502.946097] LustreError: 261634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7503.207962] Lustre: server umount lustre-MDT0001 complete [ 7513.347976] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 7516.038337] Key type lgssc unregistered [ 7516.387137] LNet: 265963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7516.396139] LNetError: 265963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7516.417623] LNet: Removed LNI 192.168.206.129@tcp [ 7517.164322] Key type .llcrypt unregistered [ 7517.166857] Key type ._llcrypt unregistered [ 7532.915229] Key type ._llcrypt registered [ 7532.917208] Key type .llcrypt registered [ 7532.995725] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 7533.726951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7533.743416] alg: No test for adler32 (adler32-zlib) [ 7534.776637] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7535.072558] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 7536.728986] Key type lgssc registered [ 7537.935198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7545.554987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7545.590848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.999745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7551.706249] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7556.583413] LustreError: 267185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.619209] LustreError: 267185:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7561.716752] LustreError: 267184:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.869790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.607414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7569.649857] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7577.270974] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7581.619799] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7588.578529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7588.881290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7592.003637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7595.595670] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7602.522984] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7617.505261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7617.514594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.536822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7618.531196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7618.533291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.502692] Lustre: server umount lustre-OST0000 complete [ 7628.772209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.776328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.790111] Lustre: Skipped 1 previous similar message [ 7628.809296] Lustre: Skipped 2 previous similar messages [ 7632.098195] Lustre: server umount lustre-MDT0000 complete [ 7633.897138] LustreError: 267185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.915168] LustreError: 267185:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7635.268512] LustreError: 267165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861724 with bad export cookie 7042180072565541131 [ 7635.275147] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.281364] LustreError: 267165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7635.699358] Lustre: server umount lustre-MDT0001 complete [ 7646.140833] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 7648.529582] Key type lgssc unregistered [ 7648.745720] LNet: 270867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7648.752831] LNetError: 270867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7648.771056] LNet: Removed LNI 192.168.206.129@tcp [ 7649.536636] Key type .llcrypt unregistered [ 7649.538524] Key type ._llcrypt unregistered [ 7670.465121] Key type ._llcrypt registered [ 7670.466828] Key type .llcrypt registered [ 7670.532715] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 7671.602583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7671.627760] alg: No test for adler32 (adler32-zlib) [ 7672.567123] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7672.729191] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 7674.376154] Key type lgssc registered [ 7675.718709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7682.950528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7682.972034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.495894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.130816] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7694.822624] LustreError: 272283:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7694.857839] LustreError: 272283:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7698.030641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.412216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7702.257799] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7709.674390] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.653260] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7721.593595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.840716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.795340] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7730.042199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7735.076400] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7759.846718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7759.858473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7759.870272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7760.870915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7760.873827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7760.879284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.901771] Lustre: Skipped 1 previous similar message [ 7762.014625] Lustre: server umount lustre-OST0000 complete [ 7765.987387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.998647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.005310] Lustre: Skipped 1 previous similar message [ 7766.022473] Lustre: Skipped 1 previous similar message [ 7771.107233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.115207] Lustre: Skipped 1 previous similar message [ 7771.921760] Lustre: server umount lustre-MDT0000 complete [ 7775.947884] LustreError: 273020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861864 with bad export cookie 13366522412244791423 [ 7775.951024] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.959305] LustreError: 273020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7776.429943] Lustre: server umount lustre-MDT0001 complete [ 7788.273540] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 7791.007256] Key type lgssc unregistered [ 7791.230691] LNet: 276593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7791.240540] LNetError: 276593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7791.263190] LNet: Removed LNI 192.168.206.129@tcp [ 7792.166401] Key type .llcrypt unregistered [ 7792.169845] Key type ._llcrypt unregistered [ 7808.243177] Key type ._llcrypt registered [ 7808.246420] Key type .llcrypt registered [ 7808.344873] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 7809.512293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7809.657086] alg: No test for adler32 (adler32-zlib) [ 7810.732181] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7810.959750] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 7812.665199] Key type lgssc registered [ 7813.961382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7820.742700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7820.769314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.271673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.346407] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7832.549097] LustreError: 277830:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.563823] LustreError: 277830:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7835.222994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.615885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.989787] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7846.571056] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.752315] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7858.126037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.511811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.556373] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 7867.805710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7871.227574] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7883.234870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.237186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.251499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.269344] Lustre: Skipped 1 previous similar message [ 7883.286841] Lustre: Skipped 1 previous similar message [ 7885.317238] Lustre: server umount lustre-OST0000 complete [ 7888.353637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.356683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.359160] Lustre: Skipped 1 previous similar message [ 7893.480026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.495375] Lustre: Skipped 2 previous similar messages [ 7894.602846] Lustre: server umount lustre-MDT0000 complete [ 7898.333973] LustreError: 277797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861987 with bad export cookie 18255927649931216032 [ 7898.334717] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7898.343258] LustreError: 277797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7898.752793] Lustre: server umount lustre-MDT0001 complete [ 7911.347608] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 7915.413448] Key type lgssc unregistered [ 7915.903523] LNet: 281535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7915.912905] LNetError: 281535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7915.936065] LNet: Removed LNI 192.168.206.129@tcp [ 7917.597191] Key type .llcrypt unregistered [ 7917.598619] Key type ._llcrypt unregistered [ 7933.244770] Key type ._llcrypt registered [ 7933.247438] Key type .llcrypt registered [ 7933.418575] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7955.218983] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 7972.162354] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 7973.756918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7973.777430] alg: No test for adler32 (adler32-zlib) [ 7974.861420] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 7975.084248] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 7976.752214] Key type lgssc registered [ 7978.472438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8018.542727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8018.582704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.909715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8019.948831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8020.070981] Lustre: lustre-MDT0000: new disk, initializing [ 8020.182913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.206600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8024.880958] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8035.398747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.588491] Lustre: 285091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8035.661978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8035.666070] Lustre: Skipped 1 previous similar message [ 8035.759215] Lustre: lustre-MDT0001: new disk, initializing [ 8035.888640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8035.929849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8035.945061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8040.305464] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8049.557542] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8055.460248] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8062.351982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.618188] Lustre: lustre-OST0000: new disk, initializing [ 8062.621335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8062.629382] Lustre: 287060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8062.712181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.974955] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8070.744611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8070.755165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8070.889476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8078.934740] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8084.684132] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8084.984978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8089.905579] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8090.142408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8096.744368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.747338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8096.780951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.788262] LustreError: Skipped 1 previous similar message [ 8096.806722] Lustre: Skipped 1 previous similar message [ 8098.265587] Lustre: server umount lustre-OST0000 complete [ 8102.378699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8102.391796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8102.405725] Lustre: Skipped 1 previous similar message [ 8102.409701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8106.981535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.982860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8107.019486] Lustre: Skipped 1 previous similar message [ 8108.752567] Lustre: server umount lustre-MDT0000 complete [ 8112.182954] LustreError: 288601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862201 with bad export cookie 15819016240182777286 [ 8112.193976] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8112.197433] LustreError: 288601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8112.661880] Lustre: server umount lustre-MDT0001 complete [ 8121.011888] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:23:28 (1787862208) [ 8128.606121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.118907] LustreError: 289472:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.138522] LustreError: 289472:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8129.199875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.806587] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8134.625844] LustreError: 289473:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.749718] LustreError: 289472:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8141.977051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.317177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8147.520848] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8154.604760] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8160.012932] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8166.664051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.969949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.069344] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8181.895288] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8193.512977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8193.520881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.544698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8196.872371] Lustre: server umount lustre-OST0000 complete [ 8203.758823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8203.768971] LustreError: Skipped 1 previous similar message [ 8203.773568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.782034] Lustre: Skipped 1 previous similar message [ 8203.786336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.791716] Lustre: Skipped 2 previous similar messages [ 8207.295325] Lustre: server umount lustre-MDT0000 complete [ 8209.889244] LustreError: 289473:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.903521] LustreError: 289473:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8211.598777] LustreError: 289453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862300 with bad export cookie 15819016240182778161 [ 8211.609889] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8211.614936] LustreError: 289453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8212.185569] Lustre: server umount lustre-MDT0001 complete [ 8226.952159] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 8230.364817] Key type lgssc unregistered [ 8230.669648] LNet: 293012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8230.678150] LNetError: 293012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8230.694368] LNet: Removed LNI 192.168.206.129@tcp [ 8231.793360] Key type .llcrypt unregistered [ 8231.794776] Key type ._llcrypt unregistered [ 8263.606228] Key type ._llcrypt registered [ 8263.608797] Key type .llcrypt registered [ 8263.681525] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 8280.608433] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 8282.192612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8282.255524] alg: No test for adler32 (adler32-zlib) [ 8283.314816] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 8283.640747] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 8285.320252] Key type lgssc registered [ 8286.500539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8328.432369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8328.468327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.745334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8329.788850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8329.932582] Lustre: lustre-MDT0000: new disk, initializing [ 8330.002806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.027423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8334.366873] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8345.213383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8345.425647] Lustre: 296838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8345.477975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8345.481868] Lustre: Skipped 1 previous similar message [ 8345.561752] Lustre: lustre-MDT0001: new disk, initializing [ 8345.649628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8345.677472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8345.694274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8350.484994] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8359.487825] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8365.191549] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8371.577919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.916468] Lustre: lustre-OST2710: new disk, initializing [ 8371.929563] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8371.942330] Lustre: 298809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8372.075649] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8377.837963] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8381.539735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8381.549297] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8381.661885] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8386.572241] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8393.981187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.259673] Lustre: lustre-OST03e8: new disk, initializing [ 8394.267818] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8394.276644] Lustre: 299873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8394.397570] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8400.201924] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8403.044133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8403.054982] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8403.171112] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8406.649575] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8413.268775] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8413.471164] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8418.033332] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8418.361557] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8422.640250] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8422.810607] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8427.130904] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8427.350935] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8432.613874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8432.629368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.662988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8434.153072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8434.154977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8434.169858] Lustre: Skipped 1 previous similar message [ 8434.192234] Lustre: Skipped 3 previous similar messages [ 8434.338855] Lustre: server umount lustre-MDT0000 complete [ 8437.941983] LustreError: 298808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862526 with bad export cookie 15957268550643356122 [ 8437.948560] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8437.953074] LustreError: 298808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8438.371936] Lustre: server umount lustre-MDT0001 complete [ 8453.294661] Lustre: server umount lustre-OST2710 complete [ 8456.096547] Lustre: 294638:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862528/real 1787862528] req@00000000abc30a53 x1874709574647424/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8456.152841] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.169762] Lustre: Skipped 1 previous similar message [ 8457.881543] Lustre: server umount lustre-OST03e8 complete [ 8478.171266] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 8488.124325] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 8525.911172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.264435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8526.318528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8526.415680] Lustre: lustre-MDT0000: new disk, initializing [ 8526.573202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.618040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8531.527315] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8542.113126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.273965] Lustre: 305120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8542.335749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8542.339232] Lustre: Skipped 1 previous similar message [ 8542.414410] Lustre: lustre-MDT0001: new disk, initializing [ 8542.498896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8542.538858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8542.546095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8546.994718] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8558.176236] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8564.485214] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8571.693051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8572.004631] Lustre: lustre-OST0000: new disk, initializing [ 8572.008325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8572.020259] Lustre: 307091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8572.132592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.113964] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8581.203853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8581.217067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8581.312650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8589.869553] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8594.420550] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8594.659556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8598.786819] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8598.985177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8602.081135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8602.096359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8602.097062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8602.108459] Lustre: Skipped 1 previous similar message [ 8602.119234] Lustre: Skipped 1 previous similar message [ 8607.207446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8607.215622] Lustre: Skipped 1 previous similar message [ 8612.325776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8612.340867] Lustre: Skipped 1 previous similar message [ 8615.392338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8615.517627] Lustre: server umount lustre-OST0000 complete [ 8622.567449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.591178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8622.595766] Lustre: Skipped 1 previous similar message [ 8625.996468] Lustre: server umount lustre-MDT0000 complete [ 8629.577861] LustreError: 305113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862718 with bad export cookie 15957268550643357795 [ 8629.582810] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.592863] LustreError: 305113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8629.915404] Lustre: server umount lustre-MDT0001 complete [ 8638.136152] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:32:05 (1787862725) [ 8639.421751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8641.312089] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:32:08 (1787862728) [ 8645.407685] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 8673.719229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.025133] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8674.041557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8674.317721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.853854] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8687.676707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.911785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8687.965349] Lustre: 310949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8693.493463] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8716.719976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.101913] Lustre: Denying initial registration attempt from nid 192.168.206.129@tcp, specified as failover [ 8717.115453] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8717.131145] LustreError: 312400:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8717.144377] LustreError: 312400:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8717.151507] LustreError: 312400:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8717.371502] Lustre: server umount lustre-OST0000 complete [ 8717.381794] LustreError: 312400:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8725.993105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8726.010279] Lustre: Skipped 1 previous similar message [ 8726.019219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8726.025076] Lustre: Skipped 1 previous similar message [ 8730.797480] Lustre: server umount lustre-MDT0000 complete [ 8731.112746] LustreError: 312250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8731.141309] LustreError: 312250:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 8735.109112] LustreError: 310939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862824 with bad export cookie 15957268550643358684 [ 8735.129441] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8735.670650] Lustre: server umount lustre-MDT0001 complete [ 8747.070763] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 8750.026303] Key type lgssc unregistered [ 8750.376786] LNet: 313528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8750.389282] LNetError: 313528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8750.424370] LNet: Removed LNI 192.168.206.129@tcp [ 8751.378284] Key type .llcrypt unregistered [ 8751.381424] Key type ._llcrypt unregistered [ 8762.764832] Key type ._llcrypt registered [ 8762.768690] Key type .llcrypt registered [ 8762.863441] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 8763.751987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8763.945445] alg: No test for adler32 (adler32-zlib) [ 8765.042758] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 8765.382582] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 8767.120157] Key type lgssc registered [ 8768.441190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8818.415761] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 8827.799493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8827.829585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.149151] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8829.174805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8829.324611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.513230] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8844.001991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.146796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8844.197207] Lustre: 316294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8844.495241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8849.257863] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8858.049573] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8864.336140] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8884.267481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8884.568700] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8884.794858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8891.491475] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8901.552469] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8907.233826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8907.254289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.278810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8907.747495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8907.775524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.789446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8910.679205] Lustre: server umount lustre-OST0000 complete [ 8916.452571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8916.462854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.480963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.580806] Lustre: server umount lustre-MDT0000 complete [ 8924.541808] LustreError: 318162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863013 with bad export cookie 4517636786822610160 [ 8924.551077] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8924.554916] LustreError: 318162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8925.224129] Lustre: server umount lustre-MDT0001 complete [ 8936.771772] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 8939.384522] Key type lgssc unregistered [ 8939.687574] LNet: 319870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8939.697088] LNetError: 319870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8939.711731] LNet: Removed LNI 192.168.206.129@tcp [ 8940.647785] Key type .llcrypt unregistered [ 8940.652422] Key type ._llcrypt unregistered [ 8963.263919] Key type ._llcrypt registered [ 8963.267203] Key type .llcrypt registered [ 8963.382144] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 8964.530980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8964.685172] alg: No test for adler32 (adler32-zlib) [ 8965.777088] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 8966.065428] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 8967.840344] Key type lgssc registered [ 8968.932563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8975.707540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8975.732622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.259521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.388574] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 8987.622446] LustreError: 321288:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8987.651208] LustreError: 321288:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8989.339819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.748973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8993.881976] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9001.741079] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9008.991575] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9019.186346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.757921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9027.373483] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9037.813438] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.608018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.630171] Lustre: Skipped 1 previous similar message [ 9050.630505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.657333] Lustre: Skipped 2 previous similar messages [ 9052.736072] Lustre: server umount lustre-MDT0000 complete [ 9055.713750] LustreError: 321288:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9055.758884] LustreError: 321288:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9057.357643] LustreError: 321268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863146 with bad export cookie 4004713660815516990 [ 9057.364121] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9057.371766] LustreError: 321268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9060.839814] LustreError: 321283:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.872158] LustreError: 321283:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9060.882166] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.898807] Lustre: Skipped 1 previous similar message [ 9060.913517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9060.925468] Lustre: Skipped 1 previous similar message [ 9062.885102] LustreError: 323780:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9062.885184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9062.910486] LustreError: 323780:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9062.932673] Lustre: Skipped 1 previous similar message [ 9064.430692] Lustre: server umount lustre-MDT0001 complete [ 9071.827201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9082.235730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.089075] LustreError: 323119:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.150141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9103.218624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9107.977882] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9112.552854] LustreError: 323120:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.576232] LustreError: 323120:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9116.554983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.992476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9117.992980] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9121.631948] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9129.450765] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9130.939789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9136.239776] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9137.781027] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9143.779982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9143.799306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9143.804734] Lustre: Skipped 1 previous similar message [ 9148.899532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9148.912495] Lustre: Skipped 1 previous similar message [ 9149.722647] Lustre: server umount lustre-OST0000 complete [ 9154.018960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.042153] Lustre: Skipped 1 previous similar message [ 9159.140396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9159.147392] Lustre: Skipped 2 previous similar messages [ 9160.176544] Lustre: server umount lustre-MDT0000 complete [ 9164.004641] LustreError: 321270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863252 with bad export cookie 4004713660815524683 [ 9164.015035] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9164.018072] LustreError: 321270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9164.740487] Lustre: server umount lustre-MDT0001 complete [ 9177.250967] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 9180.321292] Key type lgssc unregistered [ 9180.651595] LNet: 327563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9180.659033] LNetError: 327563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9180.677663] LNet: Removed LNI 192.168.206.129@tcp [ 9181.738244] Key type .llcrypt unregistered [ 9181.740957] Key type ._llcrypt unregistered [ 9207.504261] Key type ._llcrypt registered [ 9207.506617] Key type .llcrypt registered [ 9207.604274] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 9208.381382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9208.416171] alg: No test for adler32 (adler32-zlib) [ 9209.683546] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 9210.010679] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 9211.728580] Key type lgssc registered [ 9213.048741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9220.558170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9220.571414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9222.043474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.394552] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9231.330546] LustreError: 328978:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.350084] LustreError: 328978:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9234.933330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.477407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9240.666571] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9249.424196] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9255.478277] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9258.129505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9258.141284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9258.146145] Lustre: Skipped 1 previous similar message [ 9258.163354] Lustre: Skipped 1 previous similar message [ 9260.197580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9260.208790] Lustre: Skipped 1 previous similar message [ 9264.442442] Lustre: server umount lustre-MDT0000 complete [ 9265.634127] LustreError: 329705:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.647711] LustreError: 329705:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9267.466497] LustreError: 328957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863356 with bad export cookie 6699478412109585634 [ 9267.475418] LustreError: 328957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9267.478222] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.860993] Lustre: server umount lustre-MDT0001 complete [ 9280.802717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.155141] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9281.184875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9281.373085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.815784] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9293.895903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.963676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9293.986972] Lustre: 331567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9294.215625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9298.934771] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9306.539155] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9313.483854] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9325.228108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9325.613616] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9325.626345] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9325.642552] LustreError: 333453:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9325.661040] LustreError: 333453:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9325.670853] LustreError: 333453:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9325.806411] Lustre: server umount lustre-OST0000 complete [ 9325.812468] LustreError: 333453:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9336.566347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9336.934642] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9337.186111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.017988] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9344.162814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9351.535279] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9359.401236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9359.495331] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9359.605926] Lustre: lustre-OST0001: new disk, initializing [ 9359.609475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9359.614383] Lustre: 334992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9359.710981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9366.829155] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9368.664733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9368.685039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9368.818298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9376.732696] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9384.428567] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9384.434217] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9384.449471] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9384.463946] Lustre: Skipped 1 previous similar message [ 9384.476436] Lustre: Skipped 1 previous similar message [ 9386.533358] Lustre: server umount lustre-OST0001 complete [ 9389.539731] LustreError: 333920:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.586702] LustreError: 333920:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9390.049920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9390.062321] LustreError: Skipped 1 previous similar message [ 9390.074102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.118795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.684382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.688605] Lustre: Skipped 1 previous similar message [ 9394.693265] LustreError: 333921:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.725571] LustreError: 333921:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9395.967428] Lustre: server umount lustre-OST0000 complete [ 9399.777356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9399.802839] Lustre: Skipped 2 previous similar messages [ 9404.899664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.905276] Lustre: Skipped 3 previous similar messages [ 9406.259755] Lustre: server umount lustre-MDT0000 complete [ 9410.021794] LustreError: 331578:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.052796] LustreError: 331578:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9410.132528] LustreError: 331559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863499 with bad export cookie 6699478412109586187 [ 9410.134469] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9410.151765] LustreError: 331559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9410.922718] Lustre: server umount lustre-MDT0001 complete [ 9424.363470] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [ 9427.510529] Key type lgssc unregistered [ 9427.829633] LNet: 336953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9427.842860] LNetError: 336953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9427.866965] LNet: Removed LNI 192.168.206.129@tcp [ 9428.753790] Key type .llcrypt unregistered [ 9428.762693] Key type ._llcrypt unregistered [ 9500.815820] Key type ._llcrypt registered [ 9500.822480] Key type .llcrypt registered [ 9500.921104] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 9515.254162] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 9516.656969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9516.670362] alg: No test for adler32 (adler32-zlib) [ 9517.679851] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [ 9517.946955] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 9519.616524] Key type lgssc registered [ 9520.789948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9553.516877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9553.565897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.903682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9554.953339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9555.082686] Lustre: lustre-MDT0000: new disk, initializing [ 9555.193404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.208377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9559.998132] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9570.974268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.135258] Lustre: 341891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9571.183850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9571.187689] Lustre: Skipped 1 previous similar message [ 9571.322583] Lustre: lustre-MDT0001: new disk, initializing [ 9571.504486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9571.549773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9571.563351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9577.086514] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9587.624656] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9592.893216] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9600.713437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9601.136922] Lustre: lustre-OST0000: new disk, initializing [ 9601.140942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9601.146683] Lustre: 343859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9601.266661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9607.206115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9607.212357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9607.352253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9607.850980] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9618.469198] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9624.720656] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9624.954599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9629.439976] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9629.623514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9633.262023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9633.263096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9633.292395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9633.299959] Lustre: Skipped 1 previous similar message [ 9638.120901] Lustre: server umount lustre-OST0000 complete [ 9643.083569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9643.107972] Lustre: Skipped 1 previous similar message [ 9643.116400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9643.123783] Lustre: Skipped 1 previous similar message [ 9648.097882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9648.108487] Lustre: Skipped 1 previous similar message [ 9649.585471] Lustre: server umount lustre-MDT0000 complete [ 9653.672536] LustreError: 342695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863742 with bad export cookie 4297862966909320854 [ 9653.674284] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9653.683363] LustreError: 342695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9654.084653] Lustre: server umount lustre-MDT0001 complete [ 9662.948081] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:49:09 (1787863749) [ 9670.025887] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:49:17 (1787863757) [ 9677.752140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.125217] LustreError: 346613:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.139339] LustreError: 346613:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9678.181170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.395838] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9683.426371] LustreError: 346614:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.562292] LustreError: 346613:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9690.309816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.662537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9694.875568] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9702.865601] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9708.641836] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9715.182495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9715.518093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9722.754786] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9729.818843] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9741.647384] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9743.573189] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9745.151962] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9746.891249] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9748.553105] Lustre: Failing over lustre-MDT0000 [ 9748.802469] Lustre: server umount lustre-MDT0000 complete [ 9751.523187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9751.530149] LustreError: 346608:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.544884] Lustre: Skipped 3 previous similar messages [ 9751.559740] LustreError: 346608:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9756.642846] LustreError: 346613:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.671524] LustreError: 346613:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 9766.888088] LustreError: 346608:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.899074] LustreError: 346608:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 9767.841900] Lustre: 340116:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863840/real 1787863840] req@000000003ada67c5 x1874710868598144/t0(0) o400->MGC192.168.206.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787863856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9767.871984] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9768.211883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9777.485406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.589178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9782.704927] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9782.758541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9782.806541] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9782.854212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9783.695603] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9797.105447] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9799.338917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9803.794912] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9806.401041] Lustre: Failing over lustre-MDT0000 [ 9806.724714] Lustre: server umount lustre-MDT0000 complete [ 9808.369326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9808.371688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.379407] LustreError: 350421:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.379418] LustreError: 350421:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [ 9808.388778] LustreError: Skipped 1 previous similar message [ 9808.470551] Lustre: Skipped 2 previous similar messages [ 9824.736591] Lustre: 340116:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863897/real 1787863897] req@000000005e532a83 x1874710868620800/t0(0) o400->MGC192.168.206.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787863913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9824.772928] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.208550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.978210] LustreError: 340114:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cb814bc2 x1874710868625920/t0(0) o250->MGC192.168.206.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9835.328830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9835.404081] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9839.029850] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9839.982980] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9840.642356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9840.649417] Lustre: Skipped 2 previous similar messages [ 9840.682311] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9840.717285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9849.862908] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9851.368434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9861.090940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9861.105044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9861.109565] Lustre: Skipped 1 previous similar message [ 9861.124573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9864.916757] Lustre: server umount lustre-MDT0000 complete [ 9866.222910] LustreError: 347341:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9866.246897] LustreError: 347341:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 30 previous similar messages [ 9869.528599] LustreError: 346593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863958 with bad export cookie 4297862966909323570 [ 9869.529388] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9869.543725] LustreError: 346593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9869.875975] Lustre: server umount lustre-MDT0001 complete [ 9883.475729] Lustre: server umount lustre-OST0000 complete [ 9900.181756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9900.918423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9905.811954] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9914.403859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.876635] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9926.462567] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9932.196540] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9938.636821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9944.764082] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [ 9946.101821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9952.052976] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9959.110543] Lustre: server umount lustre-OST0000 complete [ 9966.049513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9966.054481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.066716] Lustre: Skipped 1 previous similar message [ 9966.070641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9966.076605] Lustre: Skipped 2 previous similar messages [ 9972.811777] Lustre: server umount lustre-MDT0000 complete [ 9976.813762] LustreError: 353713:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9976.832676] LustreError: 353713:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 9976.930770] LustreError: 352963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864065 with bad export cookie 4297862966909324312 [ 9976.947957] LustreError: 352963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9976.951572] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9986.663863] Lustre: server umount lustre-MDT0001 complete [ 9998.488931] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [10001.615644] Key type lgssc unregistered [10002.000592] LNet: 356523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10002.013331] LNetError: 356523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10002.029079] LNet: Removed LNI 192.168.206.129@tcp [10003.145491] Key type .llcrypt unregistered [10003.148423] Key type ._llcrypt unregistered [10032.429309] Key type ._llcrypt registered [10032.434084] Key type .llcrypt registered [10032.563983] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [10033.383913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10033.396869] alg: No test for adler32 (adler32-zlib) [10034.491843] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [10034.664479] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [10036.320332] Key type lgssc registered [10037.333606] Lustre: Echo OBD driver; http://www.lustre.org/ [10045.262700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10045.281204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.779695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10051.820986] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10056.177237] LustreError: 358041:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.191499] LustreError: 358041:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10060.951533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.284983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10061.453820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10067.122561] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10075.004290] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10080.743466] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10087.454115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10087.851330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10091.960857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10093.860365] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10102.150307] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10117.604893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10117.628905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10117.648034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10119.303494] Lustre: server umount lustre-MDT0000 complete [10121.701939] LustreError: 358041:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.708349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.743025] LustreError: 358041:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10121.766454] Lustre: Skipped 1 previous similar message [10123.394320] LustreError: 358022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864212 with bad export cookie 5995041751477832042 [10123.396264] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10123.417893] LustreError: 358022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10124.464370] Lustre: server umount lustre-MDT0001 complete [10139.396382] Lustre: server umount lustre-OST0000 complete [10154.884639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10155.333074] LustreError: 361775:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10155.352777] LustreError: 361775:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10155.420910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10159.604246] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10160.615766] LustreError: 361776:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10165.734166] LustreError: 361775:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.964028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.350130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10173.139300] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10181.822035] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10187.773146] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10195.056455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10195.483601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10200.632605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10201.039416] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10207.772262] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10224.107320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10224.112814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.134022] Lustre: Skipped 2 previous similar messages [10224.164416] Lustre: Skipped 2 previous similar messages [10227.072440] Lustre: server umount lustre-MDT0000 complete [10229.220333] LustreError: 361776:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.230430] LustreError: 361776:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10230.948639] LustreError: 361757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864319 with bad export cookie 5995041751477833155 [10230.968772] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.971134] LustreError: 361757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10231.457670] Lustre: server umount lustre-MDT0001 complete [10245.346828] Lustre: server umount lustre-OST0000 complete [10260.593459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10260.985083] LustreError: 365689:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.070632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10265.861267] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10274.175581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.869795] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10287.655683] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10293.768871] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10300.678856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10301.096512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10301.112643] Lustre: Skipped 1 previous similar message [10305.244755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10308.252611] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10315.882350] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10320.868439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10320.875742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.894613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10324.963931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10324.971135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10324.977103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10325.009731] Lustre: Skipped 1 previous similar message [10326.752419] Lustre: server umount lustre-OST0000 complete [10335.202041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10335.208417] Lustre: Skipped 3 previous similar messages [10336.286364] Lustre: server umount lustre-MDT0000 complete [10339.930243] LustreError: 365670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864428 with bad export cookie 5995041751477834170 [10339.937863] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10339.939847] LustreError: 365670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10340.292362] Lustre: server umount lustre-MDT0001 complete [10351.275879] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [10354.160550] Key type lgssc unregistered [10354.549424] LNet: 369224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10354.562555] LNetError: 369224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10355.622492] LNet: Removed LNI 192.168.206.129@tcp [10356.582719] Key type .llcrypt unregistered [10356.585306] Key type ._llcrypt unregistered [10380.807630] Key type ._llcrypt registered [10380.809255] Key type .llcrypt registered [10380.903684] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [10381.549697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10381.568645] alg: No test for adler32 (adler32-zlib) [10382.817794] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [10383.086566] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [10384.808211] Key type lgssc registered [10385.789868] Lustre: Echo OBD driver; http://www.lustre.org/ [10393.740370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10393.749516] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10393.753205] LustreError: 370692:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10393.765108] LustreError: 370692:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10393.772656] LustreError: 370692:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10393.777658] LustreError: 370692:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10393.785529] LustreError: 370692:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10400.678266] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10400.685738] LustreError: 371089:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10400.713400] LustreError: 371089:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10400.724206] LustreError: 371089:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10400.732340] LustreError: 371089:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10400.749202] LustreError: 371089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10418.924905] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [10421.374944] Key type lgssc unregistered [10421.666451] LNet: 372095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10421.684471] LNetError: 372095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10421.708655] LNet: Removed LNI 192.168.206.129@tcp [10422.561548] Key type .llcrypt unregistered [10422.563490] Key type ._llcrypt unregistered [10440.732159] Key type ._llcrypt registered [10440.734868] Key type .llcrypt registered [10440.846593] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [10453.599814] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [10454.799246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10454.850941] alg: No test for adler32 (adler32-zlib) [10455.937720] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [10456.164554] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [10457.872208] Key type lgssc registered [10458.787019] Lustre: Echo OBD driver; http://www.lustre.org/ [10490.651399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10490.710486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.950643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10491.988529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10492.056188] Lustre: lustre-MDT0000: new disk, initializing [10492.120573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10492.133954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10495.829184] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10505.764435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.928306] Lustre: 375233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10505.973975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10505.979262] Lustre: Skipped 1 previous similar message [10506.089639] Lustre: lustre-MDT0001: new disk, initializing [10506.208842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10506.274543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10506.288372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10510.350234] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10518.956642] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10524.593872] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10530.844807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10531.002548] Lustre: lustre-OST0000: new disk, initializing [10531.008786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10531.013932] Lustre: 377202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10531.112423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10537.135889] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10537.541094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10537.551577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10537.657472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10545.914638] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10550.741072] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10550.925498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10555.171434] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10555.385687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10558.433940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.439981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10558.448330] Lustre: Skipped 1 previous similar message [10558.465427] Lustre: Skipped 1 previous similar message [10563.458668] Lustre: server umount lustre-OST0000 complete [10567.650103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10567.658966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.686917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10568.684167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.712333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10568.718447] Lustre: Skipped 1 previous similar message [10573.134345] Lustre: server umount lustre-MDT0000 complete [10576.414741] LustreError: 375226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864665 with bad export cookie 2857731239958537157 [10576.416992] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10576.431132] LustreError: 375226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10576.949398] Lustre: server umount lustre-MDT0001 complete [10584.264388] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10585.500312] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:04:32 (1787864672) [10591.771945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.258296] LustreError: 379661:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10592.276873] LustreError: 379661:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10592.371843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10596.489549] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10597.860880] LustreError: 379662:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.979095] LustreError: 379661:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.602808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10606.911107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10612.329957] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10622.026480] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10627.556019] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10633.143099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.377369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10637.805932] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10644.208767] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10650.476072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10650.605188] Lustre: lustre-OST0001: new disk, initializing [10650.607977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10650.614240] Lustre: 382501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10650.690586] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10651.220903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10651.227710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10651.348353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10656.254355] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10664.290849] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10668.016616] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.021231] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10668.032642] Lustre: Skipped 1 previous similar message [10668.042727] Lustre: Skipped 1 previous similar message [10673.125963] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10673.145307] Lustre: Skipped 2 previous similar messages [10673.508849] Lustre: server umount lustre-OST0001 complete [10676.141258] LustreError: 381485:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.375118] LustreError: 381486:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.396247] LustreError: 381486:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10693.614063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10693.625420] LustreError: 381484:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10693.638205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10693.641224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10693.652255] LustreError: 381484:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10693.680268] Lustre: Skipped 1 previous similar message [10695.491783] Lustre: server umount lustre-OST0000 complete [10705.078743] Lustre: server umount lustre-MDT0000 complete [10707.962328] LustreError: 382500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864796 with bad export cookie 2857731239958538039 [10707.968786] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10707.971469] LustreError: 382500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10722.272122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10722.525796] Lustre: server umount lustre-MDT0001 complete [10731.541883] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [10733.782825] Key type lgssc unregistered [10734.025141] LNet: 384489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10734.033481] LNetError: 384489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10734.044520] LNet: Removed LNI 192.168.206.129@tcp [10734.763530] Key type .llcrypt unregistered [10734.769215] Key type ._llcrypt unregistered [10778.466977] Key type ._llcrypt registered [10778.468841] Key type .llcrypt registered [10778.539842] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [10779.315609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10779.347031] alg: No test for adler32 (adler32-zlib) [10780.485159] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [10780.673414] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [10782.320267] Key type lgssc registered [10783.459462] Lustre: Echo OBD driver; http://www.lustre.org/ [10790.970707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10790.994425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.187841] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10792.203570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10792.384481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.257344] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10804.122457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.239305] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10804.273514] Lustre: 387289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10804.478183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10808.584490] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10814.621481] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10819.281554] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10823.141147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.144868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10823.154037] Lustre: Skipped 1 previous similar message [10823.170307] Lustre: Skipped 1 previous similar message [10826.838819] Lustre: server umount lustre-MDT0000 complete [10830.130068] LustreError: 387282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864919 with bad export cookie 9924344405115035918 [10830.132519] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10830.139173] LustreError: 387282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10830.417100] Lustre: server umount lustre-MDT0001 complete [10838.660885] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:08:45 (1787864925) [10843.767184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.061775] LustreError: 389801:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.078918] LustreError: 389801:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10844.121619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.882698] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10849.259316] LustreError: 389802:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.371796] LustreError: 389801:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.216187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10855.652375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10859.833397] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10865.733936] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10870.350866] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10876.836769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10877.022331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10877.267911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10884.438231] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10892.381533] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10900.537644] Lustre: Permanently deactivating lustre-OST0000 [10900.556147] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10907.692059] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10909.146766] LustreError: 392788:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10909.157829] LustreError: 392788:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10909.165055] LustreError: 392788:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10911.641403] LustreError: 392836:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10911.654767] LustreError: 392836:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10919.664645] Lustre: server umount lustre-OST0000 complete [10922.324725] LustreError: 393086:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10922.329279] LustreError: 393086:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10924.002806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.005571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10924.022987] Lustre: Skipped 1 previous similar message [10924.032785] Lustre: Skipped 1 previous similar message [10929.121734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.129754] Lustre: Skipped 1 previous similar message [10934.241931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10934.249143] Lustre: Skipped 1 previous similar message [10938.336282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10938.515922] Lustre: server umount lustre-MDT0000 complete [10939.362667] LustreError: 390527:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.377131] LustreError: 390527:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10942.033474] LustreError: 389783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865030 with bad export cookie 9924344405115036499 [10942.038149] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10942.051735] LustreError: 389783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10951.825091] Lustre: server umount lustre-MDT0001 complete [10959.855508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10963.539795] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10965.859538] LustreError: 394245:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10971.480427] Lustre: 394585:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.129@tcp [10973.248527] Lustre: server umount MGS complete [10983.769626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.273970] LustreError: 395307:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.298973] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10984.302897] Lustre: Skipped 1 previous similar message [10984.339461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.141620] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [10994.658591] LustreError: 395307:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.682863] LustreError: 395307:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10996.969753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.291188] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11001.380277] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11008.171413] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11012.832341] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11014.551643] Lustre: Permanently reactivating lustre-OST0000 [11014.561991] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11014.567150] Lustre: Skipped 2 previous similar messages [11042.784934] LustreError: 397136:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11048.370423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11048.718693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.728432] Lustre: Skipped 1 previous similar message [11053.124317] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11059.087827] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11068.395618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11068.402962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.409388] LustreError: Skipped 1 previous similar message [11068.414738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11068.463993] Lustre: Skipped 1 previous similar message [11069.695916] Lustre: server umount lustre-OST0000 complete [11073.505367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11073.520500] Lustre: Skipped 1 previous similar message [11078.630348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11078.633124] Lustre: Skipped 3 previous similar messages [11087.328143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11087.487293] Lustre: server umount lustre-MDT0000 complete [11088.870476] LustreError: 395302:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.895303] LustreError: 395302:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11091.097858] LustreError: 395288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865180 with bad export cookie 9924344405115037549 [11091.100587] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11091.123371] LustreError: 395288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11091.748408] Lustre: server umount lustre-MDT0001 complete [11102.412437] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [11105.382533] Key type lgssc unregistered [11105.697964] LNet: 399775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11105.704925] LNetError: 399775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11105.715874] LNet: Removed LNI 192.168.206.129@tcp [11106.550105] Key type .llcrypt unregistered [11106.553819] Key type ._llcrypt unregistered [11124.049703] Key type ._llcrypt registered [11124.052560] Key type .llcrypt registered [11124.153484] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [11138.819675] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [11139.714143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11139.862859] alg: No test for adler32 (adler32-zlib) [11140.871280] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [11141.062426] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [11142.696115] Key type lgssc registered [11143.659238] Lustre: Echo OBD driver; http://www.lustre.org/ [11178.081088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11178.098544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.367748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11179.424670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11179.539735] Lustre: lustre-MDT0000: new disk, initializing [11179.624889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11179.650445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11183.814245] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11193.329265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11193.464480] Lustre: 402933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11193.505172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11193.510503] Lustre: Skipped 1 previous similar message [11193.581180] Lustre: lustre-MDT0001: new disk, initializing [11193.656458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11193.703256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11193.715621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11197.862613] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11206.276783] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11210.822429] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11216.168994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.350837] Lustre: lustre-OST0000: new disk, initializing [11216.355088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11216.362129] Lustre: 404900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11216.420936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11221.439547] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11224.621380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11224.628853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11224.734637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11229.598307] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11233.911651] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11234.092513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11238.427409] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11238.604724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11240.445674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11240.465267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11245.548934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11245.550066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.589339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11246.715107] Lustre: server umount lustre-OST0000 complete [11250.659777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11250.678405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11250.690507] Lustre: Skipped 1 previous similar message [11255.265111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11255.270658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.289839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11256.737249] Lustre: server umount lustre-MDT0000 complete [11259.981867] LustreError: 402924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865348 with bad export cookie 11849747973890079842 [11259.985037] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11259.995448] LustreError: 402924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11260.387771] Lustre: server umount lustre-MDT0001 complete [11268.485300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11270.716545] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:15:57 (1787865357) [11278.141028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.593686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11283.116877] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11283.947558] LustreError: 407357:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.995928] LustreError: 407357:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11288.107262] LustreError: 407358:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11290.822790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.342933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11295.916780] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11303.433530] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11308.513066] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11315.781517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11316.129259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11323.325611] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11331.384448] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11333.428806] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11337.034790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11342.817471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11342.830848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.864972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11342.870776] Lustre: Skipped 1 previous similar message [11348.401503] Lustre: server umount lustre-OST0000 complete [11353.060973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11353.074133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.085234] Lustre: Skipped 1 previous similar message [11353.091550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11353.100586] Lustre: Skipped 2 previous similar messages [11366.368173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11366.686229] Lustre: server umount lustre-MDT0000 complete [11368.418893] LustreError: 407352:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.438283] LustreError: 407352:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11370.314867] LustreError: 408091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865459 with bad export cookie 11849747973890080717 [11370.326973] LustreError: 408091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11370.328560] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11370.702508] Lustre: server umount lustre-MDT0001 complete [11380.556465] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [11383.248699] Key type lgssc unregistered [11383.528665] LNet: 411085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11383.544323] LNetError: 411085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11383.580518] LNet: Removed LNI 192.168.206.129@tcp [11384.408633] Key type .llcrypt unregistered [11384.413145] Key type ._llcrypt unregistered [11407.415184] Key type ._llcrypt registered [11407.420107] Key type .llcrypt registered [11407.543501] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [11407.976566] Key type .llcrypt unregistered [11407.980316] Key type ._llcrypt unregistered [11425.299823] Key type ._llcrypt registered [11425.303042] Key type .llcrypt registered [11425.404583] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [11426.977350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11426.994456] alg: No test for adler32 (adler32-zlib) [11428.091840] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [11428.390422] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [11430.080190] Key type lgssc registered [11431.041328] Lustre: Echo OBD driver; http://www.lustre.org/ [11439.514509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11439.530696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.050352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11446.229709] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11450.337573] LustreError: 413304:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.359255] LustreError: 413304:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11455.460545] LustreError: 413305:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11455.681558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11456.216210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11461.608791] LustreError: 413299:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.632632] LustreError: 413299:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11462.177246] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11465.699128] LustreError: 413304:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11465.717758] LustreError: 413304:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11465.724301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11469.041191] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11470.825929] LustreError: 413301:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11470.851410] LustreError: 413301:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11475.932861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.352381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11480.439808] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11490.141290] Lustre: server umount lustre-OST0000 complete [11495.912112] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11495.916222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.929839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11496.931416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.933577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11496.949819] Lustre: Skipped 1 previous similar message [11502.050322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11504.161162] Lustre: server umount lustre-MDT0000 complete [11507.170559] Lustre: 412616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865580/real 1787865580] req@000000006a0152cf x1874712872380544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787865596 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11507.176748] LustreError: 413301:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11507.197684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11507.226962] LustreError: 413301:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11507.381651] LustreError: 413284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865596 with bad export cookie 18366372703145791585 [11507.389613] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11507.719542] Lustre: server umount lustre-MDT0001 complete [11517.042486] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [11519.727678] Key type lgssc unregistered [11519.992991] LNet: 416452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11519.999390] LNetError: 416452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11520.017028] LNet: Removed LNI 192.168.206.129@tcp [11520.809214] Key type .llcrypt unregistered [11520.814054] Key type ._llcrypt unregistered [11540.059234] Key type ._llcrypt registered [11540.061373] Key type .llcrypt registered [11540.152614] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [11541.131335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11541.165705] alg: No test for adler32 (adler32-zlib) [11542.207336] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [11542.494605] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [11544.176391] Key type lgssc registered [11545.310096] Lustre: Echo OBD driver; http://www.lustre.org/ [11553.383867] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11553.423235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11568.994297] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11574.112308] LustreError: 417822:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.129@tcp: failed processing log, type 4: rc = -110 [11600.737644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11607.293106] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11615.759894] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11623.923931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.564602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11624.631432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11630.033273] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11630.052848] LustreError: 417848:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.100310] LustreError: 417848:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11635.172072] LustreError: 418857:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.217965] LustreError: 418857:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11639.544476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.878224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11644.825272] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11652.369291] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11658.029899] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11665.381199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11665.390750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11665.424597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11665.435241] Lustre: Skipped 1 previous similar message [11670.530911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11670.546677] Lustre: Skipped 1 previous similar message [11675.619069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11675.633227] Lustre: Skipped 1 previous similar message [11679.200503] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11679.518479] Lustre: server umount lustre-OST0000 complete [11685.859771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11685.867785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11685.877300] LustreError: Skipped 1 previous similar message [11685.880355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11685.923951] Lustre: Skipped 2 previous similar messages [11689.817934] Lustre: server umount lustre-MDT0000 complete [11690.982666] LustreError: 418858:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11691.001778] LustreError: 418858:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11693.482205] LustreError: 417829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865782 with bad export cookie 10822670784776331960 [11693.495089] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11693.498755] LustreError: 417829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11694.149229] Lustre: server umount lustre-MDT0001 complete [11708.364940] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [11712.015183] Key type lgssc unregistered [11712.381134] LNet: 421397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11712.390531] LNetError: 421397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11712.420248] LNet: Removed LNI 192.168.206.129@tcp [11713.646254] Key type .llcrypt unregistered [11713.648870] Key type ._llcrypt unregistered [11743.107869] Key type ._llcrypt registered [11743.111889] Key type .llcrypt registered [11743.243254] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [11744.869887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11744.904886] alg: No test for adler32 (adler32-zlib) [11746.083712] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [11746.362591] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [11748.048379] Key type lgssc registered [11749.508772] Lustre: Echo OBD driver; http://www.lustre.org/ [11758.633791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11758.652730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.343150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11765.957354] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11770.851260] LustreError: 422815:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.881966] LustreError: 422815:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11775.858255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11775.973855] LustreError: 422816:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.316145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11781.937388] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11793.328027] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11801.278433] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11809.601401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11810.193654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11817.422460] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11826.672636] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11830.640804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11837.922923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11837.933214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.953240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.994571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.998684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.018567] Lustre: Skipped 1 previous similar message [11841.038887] Lustre: Skipped 3 previous similar messages [11842.419848] Lustre: server umount lustre-MDT0000 complete [11843.502719] LustreError: 422811:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.609141] LustreError: 422797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.29@tcp arrived at 1787865934 with bad export cookie 1036065931124704574 [11845.624416] LustreError: 422797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11846.116717] LustreError: 422812:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.137694] LustreError: 422812:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11847.672687] LustreError: 422798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865936 with bad export cookie 1036065931124703909 [11847.683525] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11851.238475] LustreError: 422816:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11851.254696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.269255] LustreError: 422816:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11851.306040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.027815] Lustre: server umount lustre-OST0000 complete [11861.552157] Lustre: server umount lustre-MDT0001 complete [11873.840798] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [11877.132161] Key type lgssc unregistered [11877.454963] LNet: 426505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11877.463461] LNetError: 426505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11877.488274] LNet: Removed LNI 192.168.206.129@tcp [11878.654223] Key type .llcrypt unregistered [11878.656787] Key type ._llcrypt unregistered [11904.691675] Key type ._llcrypt registered [11904.695834] Key type .llcrypt registered [11904.815397] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [11906.061032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11906.272463] alg: No test for adler32 (adler32-zlib) [11907.417966] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [11907.693598] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [11909.456162] Key type lgssc registered [11911.343724] Lustre: Echo OBD driver; http://www.lustre.org/ [11921.409658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11921.450451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11923.073294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11929.146562] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11932.661816] LustreError: 427923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.677183] LustreError: 427923:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11937.773511] LustreError: 427924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.874802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11940.569328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11946.059040] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11954.220397] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11960.640757] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11968.179496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.576297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11974.920469] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [11982.246233] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11992.114436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11999.685704] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.29@tcp (stopping) [12002.276704] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12002.288658] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.315400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12002.325491] Lustre: Skipped 1 previous similar message [12003.718831] Lustre: server umount lustre-MDT0001 complete [12004.790147] LustreError: 427918:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.396067] LustreError: 427919:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.422173] LustreError: 427919:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12012.518588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12012.519751] LustreError: 427924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.532433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.532445] Lustre: Skipped 1 previous similar message [12012.536331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12017.319673] Lustre: server umount lustre-OST0000 complete [12021.653342] Lustre: server umount lustre-MDT0000 complete [12036.096888] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [12039.320587] Key type lgssc unregistered [12039.647977] LNet: 431635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12039.654231] LNetError: 431635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12040.683563] LNet: Removed LNI 192.168.206.129@tcp [12041.600954] Key type .llcrypt unregistered [12041.604705] Key type ._llcrypt unregistered [12067.782363] Key type ._llcrypt registered [12067.785992] Key type .llcrypt registered [12067.894739] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [12083.152293] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12084.195963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12084.420059] alg: No test for adler32 (adler32-zlib) [12085.562606] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [12085.809270] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [12087.472311] Key type lgssc registered [12088.936529] Lustre: Echo OBD driver; http://www.lustre.org/ [12130.412637] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12146.596164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12146.609973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.868573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12147.887138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12147.979790] Lustre: lustre-MDT0000: new disk, initializing [12148.121464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.143653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12152.514757] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12167.987430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.320316] Lustre: lustre-OST0000: new disk, initializing [12168.323462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12168.326681] Lustre: Skipped 1 previous similar message [12168.330678] Lustre: 436668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12168.419150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12175.555761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12175.566606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12175.608416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12176.095910] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12187.924074] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12203.529683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.727900] Lustre: 435650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12203.802185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12203.878160] Lustre: lustre-MDT0001: new disk, initializing [12204.019414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12204.069862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12204.088359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12209.342394] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12218.691925] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12218.896329] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12223.427254] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12223.645272] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12232.164024] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12232.173729] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.180293] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12232.186457] Lustre: Skipped 1 previous similar message [12233.179247] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.29@tcp (stopping) [12233.189367] Lustre: Skipped 1 previous similar message [12235.202907] Lustre: server umount lustre-MDT0001 complete [12242.405801] LustreError: 435662:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12242.438041] LustreError: 435662:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [12247.526785] LustreError: 435657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.541862] LustreError: 435657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12252.655504] LustreError: 435657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.676484] LustreError: 435657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12257.761752] LustreError: 435663:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.763737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.770729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12257.812825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12260.298828] Lustre: server umount lustre-OST0000 complete [12262.888744] LustreError: 435662:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.437730] Lustre: server umount lustre-MDT0000 complete [12283.868825] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [12286.909757] Key type lgssc unregistered [12287.325236] LNet: 440422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12287.341930] LNetError: 440422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12287.366076] LNet: Removed LNI 192.168.206.129@tcp [12288.363156] Key type .llcrypt unregistered [12288.365225] Key type ._llcrypt unregistered [12333.108153] Key type ._llcrypt registered [12333.109592] Key type .llcrypt registered [12333.200317] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [12333.538403] Key type .llcrypt unregistered [12333.541812] Key type ._llcrypt unregistered [12350.350589] Key type ._llcrypt registered [12350.353992] Key type .llcrypt registered [12350.483371] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12351.843485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12351.958257] alg: No test for adler32 (adler32-zlib) [12353.141316] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [12353.465754] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [12355.240156] Key type lgssc registered [12356.327321] Lustre: Echo OBD driver; http://www.lustre.org/ [12382.462930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12382.493068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12383.727449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12383.756779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12383.832634] Lustre: lustre-MDT0000: new disk, initializing [12383.903975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12383.927540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12388.672722] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12400.070542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12400.268411] Lustre: 443803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12400.378527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12400.387392] Lustre: Skipped 1 previous similar message [12400.520469] Lustre: lustre-MDT0001: new disk, initializing [12400.647419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12400.682808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12400.702362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12405.838435] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12416.356642] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12423.169764] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12430.085132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12430.362188] Lustre: lustre-OST0000: new disk, initializing [12430.376398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12430.385716] Lustre: 445772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12430.501339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12436.446256] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12438.638663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12438.653661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12438.768110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12445.137396] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12454.376150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.379332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12454.402366] Lustre: Skipped 2 previous similar messages [12454.410692] Lustre: Skipped 2 previous similar messages [12458.401684] Lustre: server umount lustre-MDT0000 complete [12462.143143] LustreError: 443796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866551 with bad export cookie 16315503525042885140 [12462.155937] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12462.159410] LustreError: 443796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12462.635828] Lustre: server umount lustre-MDT0001 complete [12477.610526] Lustre: server umount lustre-OST0000 complete [12490.014829] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:36:16 (1787866576) [12499.411406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12499.956460] LustreError: 447801:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12499.972687] LustreError: 447801:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [12500.033353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12505.062679] LustreError: 447802:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12505.081155] LustreError: 447802:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12506.913756] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12509.156212] LustreError: 447801:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.289651] LustreError: 447802:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.186776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12517.516784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12522.813481] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12531.401106] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12537.957413] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12558.404556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12558.671074] Lustre: Found index 0 for lustre-OST0000, updating log [12558.867235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12564.978113] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [12570.170416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12572.634344] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12579.297710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12579.316834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12579.338535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12580.390339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12580.401921] Lustre: Skipped 2 previous similar messages [12584.697798] Lustre: server umount lustre-OST0000 complete [12601.836221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12601.838253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12601.846941] Lustre: Skipped 2 previous similar messages [12601.858903] Lustre: Skipped 1 previous similar message [12606.953107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12606.961188] Lustre: Skipped 1 previous similar message [12607.661758] Lustre: server umount lustre-MDT0000 complete [12607.979629] LustreError: 447797:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12608.002842] LustreError: 447797:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12612.222955] LustreError: 447783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866701 with bad export cookie 16315503525042886540 [12612.234256] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12612.237268] LustreError: 447783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12612.903056] Lustre: server umount lustre-MDT0001 complete [12621.848464] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:38:28 (1787866708) [12630.048540] LNet: 1 local NIs in recovery (showing 1): 192.168.206.129@tcp [13247.716511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13248.520870] LustreError: 454211:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13248.541026] LustreError: 454211:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13248.611881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13253.602311] LustreError: 454212:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13255.058350] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13258.729557] LustreError: 454211:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13264.866918] LustreError: 454212:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13264.919971] LustreError: 454212:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13266.278218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13266.802659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13272.595283] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13281.933803] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13288.748864] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13308.064980] LNet: 1 local NIs in recovery (showing 1): 192.168.206.129@tcp [13309.352430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13309.665766] Lustre: Found index 0 for lustre-OST0000, updating log [13309.994899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13316.714594] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13324.658816] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13328.044342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13328.073223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13333.444538] Lustre: server umount lustre-OST0000 complete [13338.595196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13338.601318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13338.612913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13339.685582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13340.715927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13340.725264] Lustre: Skipped 3 previous similar messages [13344.687476] Lustre: server umount lustre-MDT0000 complete [13344.802470] LustreError: 454211:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13344.813885] LustreError: 454211:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13348.793906] LustreError: 454192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867437 with bad export cookie 16315503525042887625 [13348.810556] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13348.818454] LustreError: 454192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13349.247689] Lustre: server umount lustre-MDT0001 complete [13367.458953] LNet: 1 local NIs in recovery (showing 1): 192.168.206.129@tcp [13379.731180] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [13388.898453] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [13426.102869] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:51:53 (1787867513) [13447.252390] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [13450.419481] Key type lgssc unregistered [13450.796701] LNet: 460833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13450.812920] LNetError: 460833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13571.688342] LNet: Removed LNI 192.168.206.129@tcp [13572.978839] Key type .llcrypt unregistered [13572.981187] Key type ._llcrypt unregistered [13589.929644] Key type ._llcrypt registered [13589.931563] Key type .llcrypt registered [13590.010531] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [13591.302350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13591.373721] alg: No test for adler32 (adler32-zlib) [13592.556491] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [13592.865471] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [13594.593118] Key type lgssc registered [13595.767876] Lustre: Echo OBD driver; http://www.lustre.org/ [13600.413484] LNet: Added LNI 192.168.206.129@tcp100 [8/256/0/180] [13601.422492] LNet: Added LNI 192.168.206.129@tcp101 [8/256/0/180] [13602.348533] LNet: Added LNI 192.168.206.129@tcp102 [8/256/0/180] [13603.354138] LNet: Added LNI 192.168.206.129@tcp103 [8/256/0/180] [13604.318698] LNet: Added LNI 192.168.206.129@tcp104 [8/256/0/180] [13605.367765] LNet: Added LNI 192.168.206.129@tcp105 [8/256/0/180] [13606.282995] LNet: Added LNI 192.168.206.129@tcp106 [8/256/0/180] [13607.264894] LNet: Added LNI 192.168.206.129@tcp107 [8/256/0/180] [13608.151352] LNet: Added LNI 192.168.206.129@tcp108 [8/256/0/180] [13608.985711] LNet: Added LNI 192.168.206.129@tcp109 [8/256/0/180] [13609.861925] LNet: Added LNI 192.168.206.129@tcp110 [8/256/0/180] [13610.724911] LNet: Added LNI 192.168.206.129@tcp111 [8/256/0/180] [13611.647397] LNet: Added LNI 192.168.206.129@tcp112 [8/256/0/180] [13612.690458] LNet: Added LNI 192.168.206.129@tcp113 [8/256/0/180] [13613.699911] LNet: Added LNI 192.168.206.129@tcp114 [8/256/0/180] [13614.711584] LNet: Added LNI 192.168.206.129@tcp115 [8/256/0/180] [13615.750362] LNet: Added LNI 192.168.206.129@tcp116 [8/256/0/180] [13616.837630] LNet: Added LNI 192.168.206.129@tcp117 [8/256/0/180] [13617.960788] LNet: Added LNI 192.168.206.129@tcp118 [8/256/0/180] [13618.903540] LNet: Added LNI 192.168.206.129@tcp119 [8/256/0/180] [13620.049388] LNet: Added LNI 192.168.206.129@tcp120 [8/256/0/180] [13621.200620] LNet: Added LNI 192.168.206.129@tcp121 [8/256/0/180] [13622.152830] LNet: Added LNI 192.168.206.129@tcp122 [8/256/0/180] [13623.147808] LNet: Added LNI 192.168.206.129@tcp123 [8/256/0/180] [13624.145436] LNet: Added LNI 192.168.206.129@tcp124 [8/256/0/180] [13624.995264] LNet: Added LNI 192.168.206.129@tcp125 [8/256/0/180] [13625.938789] LNet: Added LNI 192.168.206.129@tcp126 [8/256/0/180] [13626.833409] LNet: Added LNI 192.168.206.129@tcp127 [8/256/0/180] [13627.801243] LNet: Added LNI 192.168.206.129@tcp128 [8/256/0/180] [13628.728717] LNet: Added LNI 192.168.206.129@tcp129 [8/256/0/180] [13629.627598] LNet: Added LNI 192.168.206.129@tcp130 [8/256/0/180] [13630.661726] LNet: Added LNI 192.168.206.129@tcp131 [8/256/0/180] [13631.630246] LNet: Added LNI 192.168.206.129@tcp132 [8/256/0/180] [13632.456226] LNet: Added LNI 192.168.206.129@tcp133 [8/256/0/180] [13633.391362] LNet: Added LNI 192.168.206.129@tcp134 [8/256/0/180] [13634.281930] LNet: Added LNI 192.168.206.129@tcp135 [8/256/0/180] [13642.615739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13642.701607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13643.865021] Lustre: 463867:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.129@tcp: can't add NID 192.168.206.129@tcp130: rc = -75 [13644.028878] LNetError: 461326:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13644.076649] LNetError: Protocol error connecting to 192.168.206.129@tcp121 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13644.101814] LNetError: Skipped 7 previous similar messages [13644.264300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13644.379090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13644.611919] Lustre: lustre-MDT0000: new disk, initializing [13644.811680] Lustre: 463925:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.129@tcp: can't add NID 192.168.206.129@tcp130: rc = -75 [13644.842592] Lustre: 463925:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13644.874388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13644.901038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13645.094178] LNet: 2 local NIs in recovery (showing 2): 192.168.206.129@tcp129, 192.168.206.129@tcp121 [13645.098936] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13645.128944] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13645.150264] LNetError: Protocol error connecting to 192.168.206.129@tcp117 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13645.178105] LNetError: Skipped 4 previous similar messages [13651.504185] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13665.538265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13665.778821] Lustre: 463884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13665.883972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13665.897486] Lustre: Skipped 1 previous similar message [13666.097577] Lustre: lustre-MDT0001: new disk, initializing [13666.203338] Lustre: 464687:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.129@tcp: can't add NID 192.168.206.129@tcp130: rc = -75 [13666.227398] Lustre: 464687:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13666.429465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13666.506987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13666.540064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13673.381105] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13675.630586] Lustre: 465029:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.129@tcp: can't add NID 192.168.206.129@tcp130: rc = -75 [13675.639425] Lustre: 465029:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13676.833490] LNetError: 461325:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13676.848801] LNetError: 461325:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13676.867813] LNetError: Protocol error connecting to 192.168.206.129@tcp128 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13679.913652] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13679.939103] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [13679.955798] LNetError: Protocol error connecting to 192.168.206.129@tcp108 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13679.971651] LNetError: Skipped 2 previous similar messages [13684.001391] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13684.014765] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13684.019371] LNetError: Protocol error connecting to 192.168.206.129@tcp114 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13684.036943] LNetError: Skipped 5 previous similar messages [13684.737808] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13691.700076] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13692.194575] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13692.222402] LNetError: 463901:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13692.234393] LNetError: Protocol error connecting to 192.168.206.129@tcp108 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13692.252916] LNetError: Skipped 4 previous similar messages [13699.344904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13699.669871] Lustre: lustre-OST0000: new disk, initializing [13699.676132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13699.687099] Lustre: 465860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13699.758868] Lustre: 465865:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.129@tcp: can't add NID 192.168.206.129@tcp130: rc = -75 [13699.765959] Lustre: 465865:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13699.814383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13707.153883] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13708.577596] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13708.592909] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [13708.599755] LNetError: Protocol error connecting to 192.168.206.129@tcp114 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13708.612058] LNetError: Skipped 14 previous similar messages [13708.862391] Lustre: 466296:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.129@tcp: can't add NID 192.168.206.129@tcp130: rc = -75 [13708.876173] Lustre: 466296:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13708.983291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13708.994910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13709.192522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13718.239144] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13729.764073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13729.779562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13729.800491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13734.896251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13734.904432] Lustre: Skipped 1 previous similar message [13740.004400] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13740.015045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13740.027784] Lustre: Skipped 1 previous similar message [13740.244683] Lustre: server umount lustre-OST0000 complete [13741.351306] LNetError: 463895:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.129 [13741.362880] LNetError: 463895:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [13741.376298] LNetError: Protocol error connecting to 192.168.206.129@tcp126 at host 192.168.206.129:988: is it running a compatible version of Lustre? [13741.398551] LNetError: Skipped 11 previous similar messages [13745.130027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13745.167912] Lustre: Skipped 1 previous similar message [13745.184472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13745.196780] Lustre: Skipped 2 previous similar messages [13750.248195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13750.257766] Lustre: Skipped 1 previous similar message [13750.750256] Lustre: server umount lustre-MDT0000 complete [13754.493054] LustreError: 465859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867843 with bad export cookie 774920540233427187 [13754.506535] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13754.861943] Lustre: server umount lustre-MDT0001 complete [13765.105430] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [13768.566084] Key type lgssc unregistered [13769.031949] LNet: 467776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13769.037881] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13769.041824] LNet: Removed LNI 192.168.206.129@tcp [13770.080152] LNet: Removed LNI 192.168.206.129@tcp100 [13770.104203] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13770.119898] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13770.126980] LNet: Removed LNI 192.168.206.129@tcp101 [13771.169642] LNet: Removed LNI 192.168.206.129@tcp102 [13771.183340] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13771.192982] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13771.216386] LNet: Removed LNI 192.168.206.129@tcp103 [13772.257960] LNet: Removed LNI 192.168.206.129@tcp104 [13773.280242] LNet: Removed LNI 192.168.206.129@tcp105 [13773.291020] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13773.305687] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13773.331837] LNet: Removed LNI 192.168.206.129@tcp106 [13774.368479] LNet: Removed LNI 192.168.206.129@tcp107 [13775.392186] LNet: Removed LNI 192.168.206.129@tcp108 [13776.417832] LNet: Removed LNI 192.168.206.129@tcp109 [13777.440195] LNet: Removed LNI 192.168.206.129@tcp110 [13777.444198] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13777.465932] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13777.481614] LNet: Removed LNI 192.168.206.129@tcp111 [13778.528280] LNet: Removed LNI 192.168.206.129@tcp112 [13779.552196] LNet: Removed LNI 192.168.206.129@tcp113 [13780.576223] LNet: Removed LNI 192.168.206.129@tcp114 [13781.600846] LNet: Removed LNI 192.168.206.129@tcp115 [13782.624309] LNet: Removed LNI 192.168.206.129@tcp116 [13783.648220] LNet: Removed LNI 192.168.206.129@tcp117 [13784.672293] LNet: Removed LNI 192.168.206.129@tcp118 [13785.696519] LNet: Removed LNI 192.168.206.129@tcp119 [13785.704551] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13785.729304] LNetError: 467776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13785.742943] LNet: Removed LNI 192.168.206.129@tcp120 [13786.848131] LNet: Removed LNI 192.168.206.129@tcp121 [13787.872248] LNet: Removed LNI 192.168.206.129@tcp122 [13788.896825] LNet: Removed LNI 192.168.206.129@tcp123 [13789.984147] LNet: Removed LNI 192.168.206.129@tcp124 [13791.009707] LNet: Removed LNI 192.168.206.129@tcp125 [13792.096485] LNet: Removed LNI 192.168.206.129@tcp126 [13793.120924] LNet: Removed LNI 192.168.206.129@tcp127 [13794.145486] LNet: Removed LNI 192.168.206.129@tcp128 [13795.168917] LNet: Removed LNI 192.168.206.129@tcp129 [13795.173355] LNet: Removed LNI 192.168.206.129@tcp130 [13795.185594] LNet: Removed LNI 192.168.206.129@tcp131 [13795.196489] LNet: Removed LNI 192.168.206.129@tcp132 [13795.204751] LNet: Removed LNI 192.168.206.129@tcp133 [13795.211386] LNet: Removed LNI 192.168.206.129@tcp134 [13795.223507] LNet: Removed LNI 192.168.206.129@tcp135 [13797.143955] Key type .llcrypt unregistered [13797.150943] Key type ._llcrypt unregistered [13826.873560] Key type ._llcrypt registered [13826.876374] Key type .llcrypt registered [13826.980479] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [13843.724212] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [13844.958953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13845.038709] alg: No test for adler32 (adler32-zlib) [13846.226611] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [13846.546113] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [13848.256229] Key type lgssc registered [13849.522458] Lustre: Echo OBD driver; http://www.lustre.org/ [13883.804505] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:59:30 (1787867970) [13901.941134] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [13904.864746] Key type lgssc unregistered [13905.224860] LNet: 471868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13905.230676] LNetError: 471868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13905.252643] LNet: Removed LNI 192.168.206.129@tcp [13906.008506] Key type .llcrypt unregistered [13906.010325] Key type ._llcrypt unregistered [13922.907917] Key type ._llcrypt registered [13922.909743] Key type .llcrypt registered [13923.006816] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [13924.040219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13924.060817] alg: No test for adler32 (adler32-zlib) [13925.071597] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [13925.359395] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [13927.089280] Key type lgssc registered [13928.357167] Lustre: Echo OBD driver; http://www.lustre.org/ [13937.477401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13937.530264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13938.782975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13938.840594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13938.951408] Lustre: lustre-MDT0000: new disk, initializing [13939.056993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13939.094262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13943.704535] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13954.252768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13954.411641] Lustre: 473079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13954.471212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13954.475763] Lustre: Skipped 1 previous similar message [13954.581942] Lustre: lustre-MDT0001: new disk, initializing [13954.710478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13954.743712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13954.750644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13960.150875] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [13971.568341] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13978.664429] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13986.468874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13986.920582] Lustre: lustre-OST0000: new disk, initializing [13986.930408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13986.940639] Lustre: 475060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13987.113573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13992.592784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13992.619100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13992.789160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13994.483477] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14005.160772] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14009.806761] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing lnet_if_list [14016.645360] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14016.674524] LNet: Added LNI 192.168.206.129@tcp10 [8/256/0/180] [14023.724229] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14023.772535] LNet: Added LNI 192.168.206.129@tcp11 [8/256/0/180] [14028.709771] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14028.745968] LNet: Added LNI 192.168.206.129@tcp12 [8/256/0/180] [14033.430844] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14033.454342] LNet: Added LNI 192.168.206.129@tcp13 [8/256/0/180] [14038.001374] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14038.037709] LNet: Added LNI 192.168.206.129@tcp14 [8/256/0/180] [14042.952421] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14042.985547] LNet: Added LNI 192.168.206.129@tcp15 [8/256/0/180] [14048.939385] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14048.965527] LNet: Added LNI 192.168.206.129@tcp16 [8/256/0/180] [14055.696938] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14055.715544] LNet: Added LNI 192.168.206.129@tcp17 [8/256/0/180] [14062.423170] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14062.454195] LNet: Added LNI 192.168.206.129@tcp18 [8/256/0/180] [14068.045886] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14068.072659] LNet: Added LNI 192.168.206.129@tcp19 [8/256/0/180] [14073.949655] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14073.977879] LNet: Added LNI 192.168.206.129@tcp20 [8/256/0/180] [14079.222581] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14079.244094] LNet: Added LNI 192.168.206.129@tcp21 [8/256/0/180] [14085.140309] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14085.172268] LNet: Added LNI 192.168.206.129@tcp22 [8/256/0/180] [14090.313720] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14090.337848] LNet: Added LNI 192.168.206.129@tcp23 [8/256/0/180] [14096.264690] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14096.293775] LNet: Added LNI 192.168.206.129@tcp24 [8/256/0/180] [14102.494991] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14102.534319] LNet: Added LNI 192.168.206.129@tcp25 [8/256/0/180] [14108.870800] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14108.908895] LNet: Added LNI 192.168.206.129@tcp26 [8/256/0/180] [14115.089393] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14115.122719] LNet: Added LNI 192.168.206.129@tcp27 [8/256/0/180] [14122.113338] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14122.157752] LNet: Added LNI 192.168.206.129@tcp28 [8/256/0/180] [14129.160735] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14129.223724] LNet: Added LNI 192.168.206.129@tcp29 [8/256/0/180] [14135.616161] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14135.653345] LNet: Added LNI 192.168.206.129@tcp30 [8/256/0/180] [14142.420416] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14142.459634] LNet: Added LNI 192.168.206.129@tcp31 [8/256/0/180] [14149.004606] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14149.039595] LNet: Added LNI 192.168.206.129@tcp32 [8/256/0/180] [14157.439586] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14157.516630] LNet: Added LNI 192.168.206.129@tcp33 [8/256/0/180] [14166.173341] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14166.220742] LNet: Added LNI 192.168.206.129@tcp34 [8/256/0/180] [14175.971035] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14175.990066] LNet: Added LNI 192.168.206.129@tcp35 [8/256/0/180] [14185.284607] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14185.329590] LNet: Added LNI 192.168.206.129@tcp36 [8/256/0/180] [14195.689174] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14195.740876] LNet: Added LNI 192.168.206.129@tcp37 [8/256/0/180] [14205.477144] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14205.516949] LNet: Added LNI 192.168.206.129@tcp38 [8/256/0/180] [14214.180052] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14214.218603] LNet: Added LNI 192.168.206.129@tcp39 [8/256/0/180] [14223.587348] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14223.628446] LNet: Added LNI 192.168.206.129@tcp40 [8/256/0/180] [14235.875945] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14235.900808] LNet: Added LNI 192.168.206.129@tcp41 [8/256/0/180] [14244.820923] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14244.858058] LNet: Added LNI 192.168.206.129@tcp42 [8/256/0/180] [14252.902950] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14252.946176] LNet: Added LNI 192.168.206.129@tcp43 [8/256/0/180] [14260.735613] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14260.776041] LNet: Added LNI 192.168.206.129@tcp44 [8/256/0/180] [14268.649031] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14268.683374] LNet: Added LNI 192.168.206.129@tcp45 [8/256/0/180] [14276.682579] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14276.717964] LNet: Added LNI 192.168.206.129@tcp46 [8/256/0/180] [14285.164931] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14285.196633] LNet: Added LNI 192.168.206.129@tcp47 [8/256/0/180] [14293.893411] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14293.950832] LNet: Added LNI 192.168.206.129@tcp48 [8/256/0/180] [14303.935885] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14303.962693] LNet: Added LNI 192.168.206.129@tcp49 [8/256/0/180] [14314.371671] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14314.431624] LNet: Added LNI 192.168.206.129@tcp50 [8/256/0/180] [14335.993874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14335.996976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14336.011643] Lustre: Skipped 2 previous similar messages [14336.059929] Lustre: Skipped 2 previous similar messages [14340.726794] Lustre: server umount lustre-MDT0000 complete [14344.644253] LustreError: 473072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868433 with bad export cookie 5555710134267399858 [14344.651334] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14344.656622] LustreError: 473072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14345.247452] Lustre: server umount lustre-MDT0001 complete [14362.592572] Lustre: 472351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868435/real 1787868435] req@000000006a9c422d x1874715490836480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787868451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14362.635234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14363.753304] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [14365.345207] Lustre: 472352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868438/real 1787868438] req@00000000b3b806d4 x1874715490836608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787868454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14367.715754] Lustre: 472354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868440/real 1787868440] req@0000000036cb4e7b x1874715490836736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787868456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14370.784373] Lustre: 472353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868443/real 1787868443] req@00000000ea78941d x1874715490836992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787868459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14372.098076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14375.904150] Lustre: 472351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868448/real 1787868448] req@00000000101965c1 x1874715490837120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787868464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14389.287120] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4d19d8ee840bd939 [14389.311099] Lustre: MGC192.168.206.129@tcp: Connection restored to 0@lo (at 0@lo) [14389.613087] LustreError: 475053:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14389.635186] LustreError: 475053:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [14389.661964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14391.736761] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14391.737569] LustreError: 475054:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14392.737118] LustreError: 475053:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14394.089919] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14397.939601] LustreError: 475054:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14401.757177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14402.147738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14406.513129] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14408.177288] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14413.810929] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14419.515173] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14440.933796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14440.943096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14440.964370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14442.991028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14443.006906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14443.019470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14449.123430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14449.128935] Lustre: Skipped 3 previous similar messages [14454.241437] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14454.248131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14454.280459] Lustre: Skipped 1 previous similar message [14454.751737] Lustre: server umount lustre-OST0000 complete [14459.388966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14459.415823] Lustre: Skipped 1 previous similar message [14464.481284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14464.488711] Lustre: Skipped 3 previous similar messages [14473.184145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14473.807085] Lustre: server umount lustre-MDT0000 complete [14474.722816] LustreError: 499308:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14474.750466] LustreError: 499308:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14477.833021] LustreError: 473884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868566 with bad export cookie 5555710134267402553 [14477.838764] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14477.841521] LustreError: 473884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14478.334206] Lustre: server umount lustre-MDT0001 complete [14490.255575] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [14492.970771] Key type lgssc unregistered [14493.346703] LNet: 502114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14493.356516] LNetError: 502114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14493.370104] LNet: Removed LNI 192.168.206.129@tcp [14493.380194] LNet: Removed LNI 192.168.206.129@tcp10 [14493.383903] LNet: Removed LNI 192.168.206.129@tcp11 [14493.387681] LNet: Removed LNI 192.168.206.129@tcp12 [14493.395530] LNet: Removed LNI 192.168.206.129@tcp13 [14493.399249] LNet: Removed LNI 192.168.206.129@tcp14 [14493.409383] LNet: Removed LNI 192.168.206.129@tcp15 [14493.412845] LNet: Removed LNI 192.168.206.129@tcp16 [14493.416217] LNet: Removed LNI 192.168.206.129@tcp17 [14493.421419] LNet: Removed LNI 192.168.206.129@tcp18 [14493.426128] LNet: Removed LNI 192.168.206.129@tcp19 [14493.431108] LNet: Removed LNI 192.168.206.129@tcp20 [14493.435285] LNet: Removed LNI 192.168.206.129@tcp21 [14493.437843] LNet: Removed LNI 192.168.206.129@tcp22 [14493.440797] LNet: Removed LNI 192.168.206.129@tcp23 [14493.443421] LNet: Removed LNI 192.168.206.129@tcp24 [14493.446187] LNet: Removed LNI 192.168.206.129@tcp25 [14493.449824] LNet: Removed LNI 192.168.206.129@tcp26 [14493.455073] LNet: Removed LNI 192.168.206.129@tcp27 [14493.461046] LNet: Removed LNI 192.168.206.129@tcp28 [14493.464885] LNet: Removed LNI 192.168.206.129@tcp29 [14493.467081] LNet: Removed LNI 192.168.206.129@tcp30 [14493.470851] LNet: Removed LNI 192.168.206.129@tcp31 [14493.474199] LNet: Removed LNI 192.168.206.129@tcp32 [14493.477746] LNet: Removed LNI 192.168.206.129@tcp33 [14493.480221] LNet: Removed LNI 192.168.206.129@tcp34 [14493.482661] LNet: Removed LNI 192.168.206.129@tcp35 [14493.485449] LNet: Removed LNI 192.168.206.129@tcp36 [14493.487704] LNet: Removed LNI 192.168.206.129@tcp37 [14493.490408] LNet: Removed LNI 192.168.206.129@tcp38 [14493.492855] LNet: Removed LNI 192.168.206.129@tcp39 [14493.495136] LNet: Removed LNI 192.168.206.129@tcp40 [14493.497484] LNet: Removed LNI 192.168.206.129@tcp41 [14493.499519] LNet: Removed LNI 192.168.206.129@tcp42 [14493.502115] LNet: Removed LNI 192.168.206.129@tcp43 [14493.506948] LNet: Removed LNI 192.168.206.129@tcp44 [14493.509893] LNet: Removed LNI 192.168.206.129@tcp45 [14493.512985] LNet: Removed LNI 192.168.206.129@tcp46 [14493.515702] LNet: Removed LNI 192.168.206.129@tcp47 [14493.518621] LNet: Removed LNI 192.168.206.129@tcp48 [14493.522156] LNet: Removed LNI 192.168.206.129@tcp49 [14493.531173] LNet: Removed LNI 192.168.206.129@tcp50 [14494.255908] Key type .llcrypt unregistered [14494.260373] Key type ._llcrypt unregistered [14520.397820] Key type ._llcrypt registered [14520.403964] Key type .llcrypt registered [14520.706310] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [14539.511626] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [14540.912662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14540.986556] alg: No test for adler32 (adler32-zlib) [14542.228270] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [14542.404710] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [14544.042919] Key type lgssc registered [14545.262500] Lustre: Echo OBD driver; http://www.lustre.org/ [14582.780776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14582.842203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14584.285784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14584.337070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14584.410448] Lustre: lustre-MDT0000: new disk, initializing [14584.469939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14584.487348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14591.271463] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14604.216586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14604.376314] Lustre: 505313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14604.421989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14604.426836] Lustre: Skipped 1 previous similar message [14604.517227] Lustre: lustre-MDT0001: new disk, initializing [14604.677960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14604.728168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14604.744816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14609.112357] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14618.212134] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14623.980905] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14632.296840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14632.515650] Lustre: *** cfs_fail_loc=90f, val=0*** [14632.646994] Lustre: lustre-OST0000: new disk, initializing [14632.658465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14632.670603] Lustre: 507329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14632.785830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14638.197708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14638.212867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14638.348441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14638.940408] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14649.165390] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14659.062459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14659.064053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14659.071506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14659.083030] Lustre: Skipped 1 previous similar message [14659.130273] LustreError: Skipped 1 previous similar message [14664.165106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14664.178660] Lustre: Skipped 1 previous similar message [14669.291957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14669.297594] Lustre: Skipped 2 previous similar messages [14671.328127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14671.509973] Lustre: server umount lustre-OST0000 complete [14676.449412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14676.459508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14676.479058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14679.535264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14682.733582] Lustre: server umount lustre-MDT0000 complete [14687.249404] LustreError: 505306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868776 with bad export cookie 16487034445697617440 [14687.257975] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14687.261557] LustreError: 505306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14687.660425] Lustre: server umount lustre-MDT0001 complete [14698.102304] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [14701.367633] Key type lgssc unregistered [14701.604040] LNet: 509245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14701.613623] LNetError: 509245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14701.624725] LNet: Removed LNI 192.168.206.129@tcp [14702.276908] Key type .llcrypt unregistered [14702.278638] Key type ._llcrypt unregistered [14721.040912] Key type ._llcrypt registered [14721.042592] Key type .llcrypt registered [14721.172767] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [14722.003583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14722.144319] alg: No test for adler32 (adler32-zlib) [14723.177925] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [14723.367719] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [14725.016546] Key type lgssc registered [14726.219392] Lustre: Echo OBD driver; http://www.lustre.org/ [14734.228870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14734.265348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14735.793372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14740.168938] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14746.081066] LustreError: 510516:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14746.105851] LustreError: 510516:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [14747.878797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14748.206205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14752.112836] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14759.663354] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14765.079357] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14771.690369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14771.933256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14778.372266] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14785.811082] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14796.843508] loop0: detected capacity change from 0 to 204800000 [14797.161057] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14797.184214] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14797.206820] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14803.866793] loop0: detected capacity change from 0 to 204800000 [14804.087107] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14804.117148] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14804.143125] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14810.266336] loop0: detected capacity change from 0 to 204800000 [14816.869599] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14817.068412] Lustre: fs15246-OST0000: new disk, initializing [14817.071660] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14817.076407] Lustre: 514110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14817.119123] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14824.918186] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14836.725186] loop1: detected capacity change from 0 to 204800000 [14843.308219] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14843.442584] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14843.526638] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14843.651448] Lustre: fs15246-MDT0000: new disk, initializing [14843.987630] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14844.023053] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14844.165021] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14845.221606] Lustre: 515234:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fd2bfac6 x1874716327461632/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787868950 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14848.854416] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [14850.276767] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14850.287500] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14850.345064] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14870.523653] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14870.556516] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14875.620193] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14876.782556] Lustre: server umount fs15246-MDT0000 complete [14880.738910] LustreError: 510511:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14885.866385] LustreError: 510510:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14887.080988] Lustre: server umount fs15246-OST0000 complete [14901.219386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14901.222959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14901.246045] Lustre: Skipped 1 previous similar message [14905.914433] Lustre: server umount lustre-OST0000 complete [14911.459323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14911.472112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14911.476619] Lustre: Skipped 1 previous similar message [14911.485760] Lustre: Skipped 2 previous similar messages [14915.810290] LustreError: 517577:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000003768b5c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [14915.903828] Lustre: server umount lustre-MDT0000 complete [14916.581121] LustreError: 514530:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14916.596982] LustreError: 514530:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14919.874635] LustreError: 510496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869008 with bad export cookie 8689375062396659616 [14919.876421] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14919.884257] LustreError: 510496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14920.193528] Lustre: server umount lustre-MDT0001 complete [14931.175887] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [14933.971320] Key type lgssc unregistered [14934.370554] LNet: 518332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14934.376932] LNetError: 518332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14935.403808] LNet: Removed LNI 192.168.206.129@tcp [14936.401136] Key type .llcrypt unregistered [14936.407536] Key type ._llcrypt unregistered [14955.418955] Key type ._llcrypt registered [14955.421102] Key type .llcrypt registered [14955.529993] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [14956.797974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14956.886853] alg: No test for adler32 (adler32-zlib) [14957.924476] Lustre: Lustre: Build Version: 2.17.57_83_g3de4944 [14958.226361] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [14959.896407] Key type lgssc registered [14960.900216] Lustre: Echo OBD driver; http://www.lustre.org/ [14996.661851] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [15007.513812] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [15041.030706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15041.059368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15042.338660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15042.364088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15042.420765] Lustre: lustre-MDT0000: new disk, initializing [15042.479610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15042.494923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15046.493512] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [15057.787950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15057.988834] Lustre: 522796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15058.095609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15058.103473] Lustre: Skipped 1 previous similar message [15058.108311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15058.229658] Lustre: lustre-MDT0001: new disk, initializing [15058.348440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15058.383266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15058.400729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15062.845478] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [15071.441302] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15076.289378] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15082.449176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15082.674402] Lustre: lustre-OST0000: new disk, initializing [15082.676766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15082.681562] Lustre: 524766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15082.758506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15089.020824] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug -1 all [15090.219507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15090.235654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15090.322992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15097.525653] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15101.977385] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15102.156237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15106.341765] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15106.560391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15110.634823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15110.637490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15110.649161] Lustre: Skipped 1 previous similar message [15110.662220] Lustre: Skipped 1 previous similar message [15114.596725] Lustre: server umount lustre-OST0000 complete [15119.846436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15119.850829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15119.866786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15120.872931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15124.356300] Lustre: server umount lustre-MDT0000 complete [15127.459526] LustreError: 522788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869216 with bad export cookie 16721058327166105929 [15127.465821] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15127.472793] LustreError: 522788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15127.792810] Lustre: server umount lustre-MDT0001 complete [15148.670915] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [15157.524994] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [15187.379576] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14907 sec ========= 18:21:14 (1787869274) [15188.790581] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:21:15 (1787869275) === [15192.249941] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:21:19 (1787869279) === [15229.501847] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [15232.170686] Key type lgssc unregistered [15232.434638] LNet: 531412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15232.448343] LNetError: 531412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15232.468739] LNet: Removed LNI 192.168.206.129@tcp [15233.393448] Key type .llcrypt unregistered [15233.395761] Key type ._llcrypt unregistered