[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 536283160 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003288] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008555] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013052] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.023372] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028463] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029183] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030143] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031129] *** VALIDATE tmpfs *** [ 0.032570] *** VALIDATE proc *** [ 0.033278] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039035] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042786] debug: unmapping init [mem 0xffffffffb1a59000-0xffffffffb1a60fff] [ 0.044193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045780] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047017] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053302] rcu: Hierarchical SRCU implementation. [ 0.055511] smp: Bringing up secondary CPUs ... [ 0.056664] x86: Booting SMP configuration: [ 0.057036] .... node #0, CPUs: #1 #2 #3 [ 0.060336] smp: Brought up 1 node, 4 CPUs [ 0.062020] smpboot: Max logical packages: 1 [ 0.063023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138319] node 0 deferred pages initialised in 73ms [ 0.142041] devtmpfs: initialized [ 0.143280] x86/mm: Memory block size: 128MB [ 0.145381] gcov: version magic: 0x41383552 [ 0.147335] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149355] pinctrl core: initialized pinctrl subsystem [ 0.150270] [ 0.150703] ************************************************************* [ 0.151014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152016] ** ** [ 0.153025] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154017] ** ** [ 0.155022] ** This means that this kernel is built to expose internal ** [ 0.156015] ** IOMMU data structures, which may compromise security on ** [ 0.157010] ** your system. ** [ 0.158008] ** ** [ 0.159013] ** If you see this message and you are not debugging the ** [ 0.160019] ** kernel, report this immediately to your vendor! ** [ 0.161020] ** ** [ 0.162013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163016] ************************************************************* [ 0.164710] NET: Registered protocol family 16 [ 0.165485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168412] cpuidle: using governor menu [ 0.169536] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170636] PCI: Using configuration type 1 for base access [ 0.171134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178153] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182033] cryptd: max_cpu_qlen set to 1000 [ 0.185028] ACPI: Added _OSI(Module Device) [ 0.186008] ACPI: Added _OSI(Processor Device) [ 0.187011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.188011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192891] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199565] ACPI: Interpreter enabled [ 0.201194] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202014] ACPI: Using IOAPIC for interrupt routing [ 0.204114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206346] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216054] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228250] acpiphp: Slot [2] registered [ 0.229102] acpiphp: Slot [5] registered [ 0.231132] acpiphp: Slot [6] registered [ 0.233202] acpiphp: Slot [7] registered [ 0.234154] acpiphp: Slot [8] registered [ 0.236083] acpiphp: Slot [9] registered [ 0.237114] acpiphp: Slot [10] registered [ 0.239129] acpiphp: Slot [3] registered [ 0.240229] acpiphp: Slot [4] registered [ 0.241100] acpiphp: Slot [11] registered [ 0.243106] acpiphp: Slot [12] registered [ 0.244152] acpiphp: Slot [13] registered [ 0.246169] acpiphp: Slot [14] registered [ 0.247252] acpiphp: Slot [15] registered [ 0.249120] acpiphp: Slot [16] registered [ 0.250085] acpiphp: Slot [17] registered [ 0.252119] acpiphp: Slot [18] registered [ 0.254099] acpiphp: Slot [19] registered [ 0.255113] acpiphp: Slot [20] registered [ 0.257080] acpiphp: Slot [21] registered [ 0.259141] acpiphp: Slot [22] registered [ 0.260169] acpiphp: Slot [23] registered [ 0.262099] acpiphp: Slot [24] registered [ 0.263071] acpiphp: Slot [25] registered [ 0.264188] acpiphp: Slot [26] registered [ 0.266141] acpiphp: Slot [27] registered [ 0.267096] acpiphp: Slot [28] registered [ 0.269201] acpiphp: Slot [29] registered [ 0.271060] acpiphp: Slot [30] registered [ 0.272118] acpiphp: Slot [31] registered [ 0.273158] PCI host bridge to bus 0000:00 [ 0.275022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290852] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304653] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.311738] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.314022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.316018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321758] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.324891] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.327055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.329842] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344898] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348855] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352639] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.369017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.384029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.394705] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.406017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.421019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.433207] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.439019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.444015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.461042] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.474103] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.491019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.523020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.539164] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.550020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.597021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.614106] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.628020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655036] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.667981] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.669274] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.672040] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.673236] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.675170] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.680045] iommu: Default domain type: Passthrough [ 0.682235] SCSI subsystem initialized [ 0.683119] ACPI: bus type USB registered [ 0.684000] usbcore: registered new interface driver usbfs [ 0.684000] usbcore: registered new interface driver hub [ 0.686065] usbcore: registered new device driver usb [ 0.687135] pps_core: LinuxPPS API ver. 1 registered [ 0.688008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.690061] PTP clock support registered [ 0.692154] EDAC MC: Ver: 3.0.0 [ 0.693618] PCI: Using ACPI for IRQ routing [ 0.695662] NetLabel: Initializing [ 0.696009] NetLabel: domain hash size = 128 [ 0.697011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.698070] NetLabel: unlabeled traffic allowed by default [ 0.700129] vgaarb: loaded [ 0.701227] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.702007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.709000] clocksource: Switched to clocksource kvm-clock [ 0.809603] VFS: Disk quotas dquot_6.6.0 [ 0.811263] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814412] *** VALIDATE ramfs *** [ 0.815896] *** VALIDATE hugetlbfs *** [ 0.817740] pnp: PnP ACPI init [ 0.820534] pnp: PnP ACPI: found 6 devices [ 0.842065] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.845331] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.847469] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.849682] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.851735] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.853477] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.855721] NET: Registered protocol family 2 [ 0.858388] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.862967] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.865324] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.869050] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.873175] TCP: Hash tables configured (established 65536 bind 65536) [ 0.875456] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.879113] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.882192] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885189] NET: Registered protocol family 1 [ 0.887553] RPC: Registered named UNIX socket transport module. [ 0.890187] RPC: Registered udp transport module. [ 0.891873] RPC: Registered tcp transport module. [ 0.893821] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.896124] NET: Registered protocol family 44 [ 0.897813] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.900464] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.902857] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.905434] PCI: CLS 0 bytes, default 64 [ 0.907303] Unpacking initramfs... [ 2.352201] debug: unmapping init [mem 0xffff9a45fcc54000-0xffff9a45fffbffff] [ 2.356696] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.358888] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.361906] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.887596] Initialise system trusted keyrings [ 2.888940] Key type blacklist registered [ 2.890786] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.899310] zbud: loaded [ 2.902426] *** VALIDATE nfs *** [ 2.903713] *** VALIDATE nfs4 *** [ 2.905409] pstore: using deflate compression [ 2.909461] Platform Keyring initialized [ 3.005813] NET: Registered protocol family 38 [ 3.007021] Key type asymmetric registered [ 3.008079] Asymmetric key parser 'x509' registered [ 3.008977] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.010683] io scheduler mq-deadline registered [ 3.011755] io scheduler kyber registered [ 3.012758] io scheduler bfq registered [ 3.014054] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.015921] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.017645] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.019244] ACPI: Power Button [PWRF] [ 3.022479] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.026751] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.035279] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.039982] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.050131] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.074933] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.099888] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.103246] Non-volatile memory driver v1.3 [ 3.104226] Linux agpgart interface v0.103 [ 3.132091] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.134792] vda: detected capacity change from 0 to 74715136 [ 3.149305] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.152362] vdb: detected capacity change from 0 to 1073741824 [ 3.166828] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.169410] vdc: detected capacity change from 0 to 2621440000 [ 3.182559] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.184688] vdd: detected capacity change from 0 to 2621440000 [ 3.200141] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.202886] vde: detected capacity change from 0 to 4294967296 [ 3.218526] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.220406] vdf: detected capacity change from 0 to 4294967296 [ 3.226425] libphy: Fixed MDIO Bus: probed [ 3.234623] usbcore: registered new interface driver usbserial_generic [ 3.236517] usbserial: USB Serial support registered for generic [ 3.238432] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.243087] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.245072] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.247511] mousedev: PS/2 mouse device common for all mice [ 3.250152] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.252573] rtc_cmos 00:05: RTC can wake from S4 [ 3.256054] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.256550] rtc_cmos 00:05: registered as rtc0 [ 3.263808] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.265521] intel_pstate: CPU model not supported [ 3.267082] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.269814] hid: raw HID events driver (C) Jiri Kosina [ 3.271302] usbcore: registered new interface driver usbhid [ 3.272453] usbhid: USB HID core driver [ 3.273388] drop_monitor: Initializing network drop monitor service [ 3.275037] Initializing XFRM netlink socket [ 3.276271] NET: Registered protocol family 10 [ 3.278277] Segment Routing with IPv6 [ 3.279281] NET: Registered protocol family 17 [ 3.280853] mpls_gso: MPLS GSO support [ 3.284158] RAS: Correctable Errors collector initialized. [ 3.285697] AVX version of gcm_enc/dec engaged. [ 3.286642] AES CTR mode by8 optimization enabled [ 3.343934] sched_clock: Marking stable (3343911055, 0)->(4341217433, -997306378) [ 3.346704] registered taskstats version 1 [ 3.348402] Loading compiled-in X.509 certificates [ 3.350564] zswap: loaded using pool lzo/zbud [ 3.368807] Key type big_key registered [ 3.380849] Key type encrypted registered [ 3.382281] ima: No TPM chip found, activating TPM-bypass! [ 3.383736] ima: Allocated hash algorithm: sha1 [ 3.384887] ima: No architecture policies found [ 3.386390] evm: Initialising EVM extended attributes: [ 3.387521] evm: security.selinux [ 3.388625] evm: security.ima [ 3.389482] evm: security.capability [ 3.390443] evm: HMAC attrs: 0x1 [ 3.392437] rtc_cmos 00:05: setting system clock to 2026-08-07 11:00:40 UTC (1786100440) [ 3.397357] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 3.399432] debug: unmapping init [mem 0xffffffffb1782000-0xffffffffb1a58fff] [ 3.409132] Write protecting the kernel read-only data: 28672k [ 3.411396] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 3.413447] debug: unmapping init [mem 0xffffffffb0714000-0xffffffffb07fffff] [ 3.444103] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.452436] systemd[1]: Detected virtualization kvm. [ 3.453935] systemd[1]: Detected architecture x86-64. [ 3.456108] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.481863] systemd[1]: No hostname configured. [ 3.482910] systemd[1]: Set hostname to . [ 3.484444] random: systemd: uninitialized urandom read (16 bytes read) [ 3.486052] systemd[1]: Initializing machine ID from random generator. [ 3.649162] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651408] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.657178] random: systemd: uninitialized urandom read (16 bytes read) [ 3.659472] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.663663] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.310549] device-mapper: uevent: version 1.0.3 [ 4.312843] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.062158] virtio_net virtio0 ens2: renamed from eth0 [ 5.095092] random: fast init done [ 5.171223] scsi host0: ata_piix [ 5.199622] scsi host1: ata_piix [ 5.204955] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.209394] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.695868] dracut-initqueue[596]: RTNETLINK answers: File exists [ 9.894949] random: crng init done [ 9.896685] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.383322] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.545759] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.799307] SELinux: Disabled at runtime. [ 11.855406] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.862118] systemd[1]: Detected virtualization kvm. [ 11.863327] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.338508] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.342036] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.347356] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.351853] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.355535] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.362989] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.370651] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ 12.548961] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.769326] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.060689] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.101969] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.225224] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.236537] EDAC sbridge: Ver: 1.1.2 [ 14.688523] Key type dns_resolver registered [ 15.028242] NFS: Registering the id_resolver key type [ 15.030321] Key type id_resolver registered [ 15.032033] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg250-server login: [ 42.800702] hrtimer: interrupt took 27708654 ns [ 71.069639] libcfs: loading out-of-tree module taints kernel. [ 71.143211] Key type ._llcrypt registered [ 71.145908] Key type .llcrypt registered [ 71.276447] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 93.856791] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 95.526863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.539268] alg: No test for adler32 (adler32-zlib) [ 97.212786] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 98.580424] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 100.431303] Key type lgssc registered [ 102.961342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.046808] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.060564] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 183.740772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.804059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.297257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.342755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.497077] Lustre: lustre-MDT0000: new disk, initializing [ 185.603684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.625137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.587507] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.205961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.348562] Lustre: 6532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.411191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.421276] Lustre: Skipped 1 previous similar message [ 207.563869] Lustre: lustre-MDT0001: new disk, initializing [ 207.671236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.693147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.714425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.716797] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.393942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.888073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.302467] Lustre: lustre-OST0000: new disk, initializing [ 229.306963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.311626] Lustre: 8474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.392911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.067636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.085412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.164876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.360437] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.346421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.458708] Lustre: lustre-OST0001: new disk, initializing [ 251.463674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.468143] Lustre: 9545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.519352] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.876807] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.115175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.142166] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.213926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.392791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.803447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 288.512298] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing check_logdir /tmp/testlogs/ [ 294.064983] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing yml_node [ 298.152397] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 300.974687] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 303.769342] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 305.515610] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 7 07:05:41 EDT 2026 [ 323.503654] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 324.990320] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.936932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.943783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.952064] Lustre: Skipped 1 previous similar message [ 339.971963] Lustre: Skipped 3 previous similar messages [ 345.057971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.073645] Lustre: Skipped 3 previous similar messages [ 345.843789] Lustre: server umount lustre-MDT0000 complete [ 349.893942] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786100787 with bad export cookie 5392993990893952325 [ 349.896951] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.900952] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 350.179062] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.183856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.201579] Lustre: Skipped 3 previous similar messages [ 350.222634] Lustre: Skipped 1 previous similar message [ 354.216666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 354.228484] Lustre: Skipped 1 previous similar message [ 356.543030] Lustre: server umount lustre-MDT0001 complete [ 360.861547] Lustre: server umount lustre-OST0000 complete [ 364.691202] Lustre: server umount lustre-OST0001 complete [ 372.015763] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 381.922882] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 422.687502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.875719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.895573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.977310] Lustre: lustre-MDT0000: new disk, initializing [ 423.143201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.177173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 429.465852] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 444.171245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.415196] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 444.510022] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 444.515561] Lustre: Skipped 1 previous similar message [ 444.602369] Lustre: lustre-MDT0001: new disk, initializing [ 444.714312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.761075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 444.773934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.885842] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 460.490475] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.004928] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.088426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.400801] Lustre: lustre-OST0000: new disk, initializing [ 477.406334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.417316] Lustre: 17070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.532530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.887098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 478.898525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 479.009727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.341554] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 496.680527] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 502.213688] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.497265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.886911] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 508.090122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 515.043600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.045838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.047467] LustreError: Skipped 1 previous similar message [ 515.062622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.077880] Lustre: Skipped 1 previous similar message [ 515.091203] Lustre: Skipped 1 previous similar message [ 524.255140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 524.380328] Lustre: server umount lustre-OST0000 complete [ 530.417514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.432774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.438243] Lustre: Skipped 1 previous similar message [ 530.457599] Lustre: Skipped 3 previous similar messages [ 536.062044] Lustre: server umount lustre-MDT0000 complete [ 540.647778] LustreError: 15973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.695893] LustreError: 15973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 541.649767] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786100978 with bad export cookie 5392993990893953858 [ 541.676155] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 541.679815] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.447115] Lustre: server umount lustre-MDT0001 complete [ 547.900461] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:09:43 (1786100983) [ 550.838978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 553.494632] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:09:48 (1786100988) [ 561.938921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.565288] LustreError: 19482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.619877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.791026] LustreError: 19483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.131890] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 572.896433] LustreError: 19482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.019070] LustreError: 19483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.601754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.188790] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 594.469812] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.901265] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 610.700272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 619.365279] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 628.528738] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 644.070165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.081128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.089443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 644.109692] Lustre: Skipped 3 previous similar messages [ 644.988632] Lustre: server umount lustre-OST0000 complete [ 650.720906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.304379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.327847] Lustre: Skipped 2 previous similar messages [ 656.120040] Lustre: server umount lustre-MDT0000 complete [ 659.425820] LustreError: 19482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.448445] LustreError: 19482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 660.096520] LustreError: 19462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101097 with bad export cookie 5392993990893954740 [ 660.099137] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.103220] LustreError: 19462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 660.404693] Lustre: server umount lustre-MDT0001 complete [ 672.821608] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 676.367238] Key type lgssc unregistered [ 676.742815] LNet: 23043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.749037] LNetError: 23043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.764214] LNet: Removed LNI 192.168.202.150@tcp [ 678.118216] Key type .llcrypt unregistered [ 678.120116] Key type ._llcrypt unregistered [ 704.204204] Key type ._llcrypt registered [ 704.210749] Key type .llcrypt registered [ 704.345074] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 705.813630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 705.916032] alg: No test for adler32 (adler32-zlib) [ 707.023419] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 707.264083] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 709.055741] Key type lgssc registered [ 710.791648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 721.843798] loop: module loaded [ 721.929105] loop0: detected capacity change from 0 to 204800000 [ 722.166224] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.179653] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.500961] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.997240] loop0: detected capacity change from 0 to 204800000 [ 730.253054] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.269756] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.280992] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 736.447550] loop0: detected capacity change from 0 to 204800000 [ 743.626658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 743.676156] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.053935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 745.115285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 745.258281] Lustre: lustre-MDT0000: new disk, initializing [ 745.376658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.396201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 750.660772] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 760.710768] loop1: detected capacity change from 0 to 204800000 [ 767.255910] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 767.585926] Lustre: lustre-OST07c6: new disk, initializing [ 767.593594] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 767.601266] Lustre: Skipped 1 previous similar message [ 767.608404] Lustre: 26232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 767.699320] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 774.814941] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 776.782422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 776.787838] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 776.837596] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 781.072057] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 781.088110] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 781.093405] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 781.102180] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 781.107901] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 781.114198] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 803.701242] Lustre: 26140:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 803.709919] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.718034] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.725323] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 803.736352] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 803.744660] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 809.416817] Lustre: 26140:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 809.425573] Lustre: 26140:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 809.429034] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 809.433831] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.437924] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 809.455604] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.466566] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 809.486073] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.500455] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 809.508799] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.517415] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 809.525609] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.766423] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 812.782264] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 812.798746] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 812.812905] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.824228] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.842537] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.857867] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 812.865817] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.870223] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 812.880996] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.888726] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.891433] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.408906] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 821.425176] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 821.432384] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.442340] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.449482] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.464206] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.477625] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 821.485253] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.490106] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 821.496473] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.507795] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.525203] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 832.677141] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 832.684177] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 832.689138] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 832.697858] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.702879] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.712194] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.721786] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 832.731798] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.742264] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 832.749503] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.755102] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.764048] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 882.155195] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.183156] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.130490] Lustre: server umount lustre-OST07c6 complete [ 897.464601] Lustre: server umount lustre-MDT0000 complete [ 916.675501] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 920.273673] Key type lgssc unregistered [ 920.762445] LNet: 28806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.770384] LNetError: 28806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.799276] LNet: Removed LNI 192.168.202.150@tcp [ 921.986244] Key type .llcrypt unregistered [ 921.990701] Key type ._llcrypt unregistered [ 942.106436] Key type ._llcrypt registered [ 942.110768] Key type .llcrypt registered [ 942.234693] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 943.511152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.718522] alg: No test for adler32 (adler32-zlib) [ 944.791877] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 945.001234] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 946.751244] Key type lgssc registered [ 947.961957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.196556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.239423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.877419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.992463] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 968.166325] LustreError: 30077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.185959] LustreError: 30077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 973.281502] LustreError: 30078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.736049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.199409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.823397] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 988.682653] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 995.727262] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1004.167704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.435674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.756944] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1020.476761] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1028.018836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 1050.597490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1050.602286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.623217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.618196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.619049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.656301] Lustre: Skipped 1 previous similar message [ 1055.844897] Lustre: server umount lustre-OST0000 complete [ 1061.345784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1061.352778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.383523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.983253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.997317] Lustre: Skipped 2 previous similar messages [ 1067.827933] Lustre: server umount lustre-MDT0000 complete [ 1072.005107] LustreError: 30060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101509 with bad export cookie 4136973914532060001 [ 1072.021629] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.023010] LustreError: 30060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1072.701147] Lustre: server umount lustre-MDT0001 complete [ 1085.159943] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1087.625770] Key type lgssc unregistered [ 1087.870668] LNet: 33840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.875051] LNetError: 33840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.894163] LNet: Removed LNI 192.168.202.150@tcp [ 1088.987164] Key type .llcrypt unregistered [ 1088.991687] Key type ._llcrypt unregistered [ 1113.112682] Key type ._llcrypt registered [ 1113.113962] Key type .llcrypt registered [ 1113.164588] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1114.780062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1114.806604] alg: No test for adler32 (adler32-zlib) [ 1115.876883] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1116.188890] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1117.903332] Key type lgssc registered [ 1119.286666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.257972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1130.296717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.792822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.618544] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1141.221580] LustreError: 35111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.249439] LustreError: 35111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1146.341161] LustreError: 35112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.547583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.987974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1154.169428] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1163.164350] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.217966] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.415826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.968889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.973462] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1194.519785] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1201.537412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1217.005592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.015450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1217.017797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1217.029041] Lustre: Skipped 1 previous similar message [ 1222.016989] Lustre: server umount lustre-OST0000 complete [ 1227.243980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.268528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.278693] Lustre: Skipped 1 previous similar message [ 1232.354301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.854983] Lustre: server umount lustre-MDT0000 complete [ 1237.396529] LustreError: 35091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101674 with bad export cookie 18339820215711904588 [ 1237.400769] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.407619] LustreError: 35091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1237.783636] Lustre: server umount lustre-MDT0001 complete [ 1250.154547] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1253.184753] Key type lgssc unregistered [ 1253.544829] LNet: 38672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.551240] LNetError: 38672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.568969] LNet: Removed LNI 192.168.202.150@tcp [ 1254.633668] Key type .llcrypt unregistered [ 1254.636883] Key type ._llcrypt unregistered [ 1278.257491] Key type ._llcrypt registered [ 1278.260359] Key type .llcrypt registered [ 1278.391246] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1279.494813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1279.533819] alg: No test for adler32 (adler32-zlib) [ 1280.712599] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1280.934308] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1282.623315] Key type lgssc registered [ 1283.711993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1291.864740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1291.899590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.650881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.863482] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1303.013576] LustreError: 40091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.040472] LustreError: 40091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1308.136252] LustreError: 40092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.434088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.002129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1314.433559] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1323.154352] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.324744] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1336.384380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.952022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.977941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1343.614601] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1351.602508] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1361.386568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1361.392915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.411536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.682703] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1364.982728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.672179] Lustre: server umount lustre-MDT0000 complete [ 1369.734224] LustreError: 42584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.434798] LustreError: 40827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101809 with bad export cookie 4007254398555528849 [ 1372.442671] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.451846] LustreError: 40827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1374.843910] LustreError: 40087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.861027] LustreError: 40087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1374.867620] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (stopping) [ 1374.879782] Lustre: Skipped 3 previous similar messages [ 1375.202876] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.219593] Lustre: Skipped 1 previous similar message [ 1377.249561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1377.277867] Lustre: Skipped 1 previous similar message [ 1378.833763] Lustre: server umount lustre-MDT0001 complete [ 1393.763878] Lustre: server umount lustre-OST0000 complete [ 1408.504896] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1411.028589] Key type lgssc unregistered [ 1411.298028] LNet: 43957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.302259] LNetError: 43957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.335964] LNet: Removed LNI 192.168.202.150@tcp [ 1413.249346] Key type .llcrypt unregistered [ 1413.252290] Key type ._llcrypt unregistered [ 1436.448741] Key type ._llcrypt registered [ 1436.453467] Key type .llcrypt registered [ 1436.564596] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1437.713129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1437.943949] alg: No test for adler32 (adler32-zlib) [ 1438.960982] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1439.161925] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1440.831206] Key type lgssc registered [ 1442.231908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1450.795804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1450.808994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.351504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.954502] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1462.765074] LustreError: 45369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.803397] LustreError: 45369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1467.057686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.447767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.670543] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1482.278660] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1488.738346] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1496.808555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.152821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.203975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1504.113569] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1513.460899] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1520.295235] Lustre: lustre-OST0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1522.657384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.670608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.672671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.682395] Lustre: Skipped 1 previous similar message [ 1522.692141] Lustre: Skipped 1 previous similar message [ 1525.380707] Lustre: lustre-OST0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1525.915976] Lustre: server umount lustre-OST0000 complete [ 1547.236138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.241563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.250569] Lustre: Skipped 1 previous similar message [ 1547.260315] Lustre: Skipped 1 previous similar message [ 1551.465852] Lustre: server umount lustre-MDT0000 complete [ 1552.357452] LustreError: 46098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.389755] LustreError: 46098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1555.566613] LustreError: 47948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101992 with bad export cookie 6596658386597670924 [ 1555.569508] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.577765] LustreError: 47948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1562.106614] Lustre: server umount lustre-MDT0001 complete [ 1573.654335] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1577.540539] Key type lgssc unregistered [ 1577.840817] LNet: 49084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.857821] LNetError: 49084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.888781] LNet: Removed LNI 192.168.202.150@tcp [ 1579.084575] Key type .llcrypt unregistered [ 1579.086860] Key type ._llcrypt unregistered [ 1605.345889] Key type ._llcrypt registered [ 1605.349306] Key type .llcrypt registered [ 1605.434290] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1606.598227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1606.869843] alg: No test for adler32 (adler32-zlib) [ 1608.099451] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1608.408815] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1610.087315] Key type lgssc registered [ 1611.438872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.238723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.257306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.791203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.325343] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1630.188142] LustreError: 50502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.201360] LustreError: 50502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1635.298102] LustreError: 50503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.984028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.499509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1641.884220] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1650.911848] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1657.855988] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1666.000933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1666.401123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.786947] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1682.164392] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1690.044558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1698.569931] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1702.363396] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1715.852690] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-07 7h29m10s [ 1717.896852] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1719.961425] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1720.802755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.808427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.818198] Lustre: Skipped 2 previous similar messages [ 1720.835180] Lustre: Skipped 2 previous similar messages [ 1721.908714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.917580] Lustre: Skipped 1 previous similar message [ 1726.074205] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1726.082389] Lustre: Skipped 2 previous similar messages [ 1732.240137] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1732.243905] Lustre: Skipped 3 previous similar messages [ 1734.111152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1734.334497] Lustre: server umount lustre-MDT0000 complete [ 1738.048043] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1738.365456] LustreError: 50498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.518525] LustreError: 52341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.551336] LustreError: 52341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1747.142849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.246153] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.611314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.672723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1748.667758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1753.600389] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1758.225162] Lustre: DEBUG MARKER: Wait for df (36768) ... [ 1759.755303] Lustre: DEBUG MARKER: done [ 1764.321230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.347880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1764.354204] Lustre: Skipped 1 previous similar message [ 1764.378314] Lustre: Skipped 4 previous similar messages [ 1770.423576] Lustre: server umount lustre-OST0000 complete [ 1779.682289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1779.683125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.725645] Lustre: Skipped 1 previous similar message [ 1780.771195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.775872] Lustre: Skipped 5 previous similar messages [ 1781.836823] Lustre: server umount lustre-MDT0000 complete [ 1786.620291] LustreError: 50482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102223 with bad export cookie 3973557418324585269 [ 1786.639577] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1787.195144] Lustre: server umount lustre-MDT0001 complete [ 1801.051329] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1804.489226] Key type lgssc unregistered [ 1804.853334] LNet: 55451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.859782] LNetError: 55451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.933459] LNet: Removed LNI 192.168.202.150@tcp [ 1807.108357] Key type .llcrypt unregistered [ 1807.110543] Key type ._llcrypt unregistered [ 1852.180643] Key type ._llcrypt registered [ 1852.182795] Key type .llcrypt registered [ 1852.289186] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1853.511461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1853.557404] alg: No test for adler32 (adler32-zlib) [ 1854.822598] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 1855.183361] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1856.951231] Key type lgssc registered [ 1858.256693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1867.340648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1867.359577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.747461] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1868.764840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1868.911939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.911208] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1883.776141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.867621] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1883.912418] Lustre: 57661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.203604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.877157] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1900.241652] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.785587] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1917.080916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1917.184900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1917.341735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.420633] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1927.337859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1932.228983] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1939.984229] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-07 7h32m56s [ 1941.973134] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1951.621184] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1955.693848] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1955.788470] LustreError: 57668:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1975.996615] Lustre: lustre-MDT0000: Client 90029db6-05b7-49e3-a52f-170b1158c866 (at 192.168.202.50@tcp) reconnecting [ 1995.823177] LustreError: 57668:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1998.201757] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2001.308772] Lustre: DEBUG MARKER: done [ 2009.056364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.059845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.095227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.544808] Lustre: server umount lustre-OST0000 complete [ 2014.698315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.702925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.725262] Lustre: Skipped 1 previous similar message [ 2015.843398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.856177] Lustre: Skipped 2 previous similar messages [ 2020.911224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.915040] Lustre: Skipped 1 previous similar message [ 2024.187287] Lustre: server umount lustre-MDT0000 complete [ 2028.957220] LustreError: 61240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102466 with bad export cookie 4160642005077655791 [ 2028.962403] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.967653] LustreError: 61240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.447405] Lustre: server umount lustre-MDT0001 complete [ 2042.042427] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2044.553956] Key type lgssc unregistered [ 2044.805848] LNet: 61995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.810651] LNetError: 61995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.827065] LNet: Removed LNI 192.168.202.150@tcp [ 2046.057263] Key type .llcrypt unregistered [ 2046.066300] Key type ._llcrypt unregistered [ 2101.984729] Key type ._llcrypt registered [ 2101.992259] Key type .llcrypt registered [ 2102.227093] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2104.417167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.460124] alg: No test for adler32 (adler32-zlib) [ 2105.734597] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2106.303566] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2108.042637] Key type lgssc registered [ 2110.004368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.641669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2114.655755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.108862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2116.158114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2116.359734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.573739] Lustre: Failing over lustre-MDT0000 [ 2118.026782] Lustre: server umount lustre-MDT0000 complete [ 2126.200504] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:36:01 (1786102561) [ 2132.950311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.813077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.832588] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2151.661284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.818182] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2151.880307] Lustre: 64794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2152.283603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.298212] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2166.768932] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.753342] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.898996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2180.095167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2180.254133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.802478] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2187.436482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 2195.869996] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.290662] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2213.346209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.367812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.372819] Lustre: Skipped 1 previous similar message [ 2213.401167] Lustre: Skipped 2 previous similar messages [ 2216.003946] Lustre: server umount lustre-MDT0000 complete [ 2222.106308] LustreError: 65531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102659 with bad export cookie 4985449729219143984 [ 2222.135355] LustreError: 65531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2222.136943] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.134929] Lustre: server umount lustre-MDT0001 complete [ 2230.211853] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2237.069299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2238.945169] Lustre: 63719:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102660/real 1786102660] req@00000000de9f66dc x1872864260171136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786102676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2239.022251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.051248] Lustre: Skipped 1 previous similar message [ 2243.167439] Lustre: 63719:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102664/real 1786102664] req@00000000c082be81 x1872864260171264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786102680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.116126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.223934] Lustre: 63721:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102669/real 1786102669] req@000000006d086307 x1872864260171648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786102685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.237097] Lustre: 63721:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2254.815483] Lustre: 63719:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102675/real 1786102675] req@00000000fa9de93b x1872864260171776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786102691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.258065] LustreError: 66646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.277178] LustreError: 66646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2268.386985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.542683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2269.440761] LustreError: 66647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.452283] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2273.963128] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2274.787288] LustreError: 66646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.907212] LustreError: 66647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.004814] LustreError: 68568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.032772] LustreError: 68568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2284.178779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.507377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2289.573962] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2289.650029] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2299.596297] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2307.190447] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2320.359129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.374106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.374310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.392212] Lustre: Skipped 2 previous similar messages [ 2321.549282] Lustre: server umount lustre-MDT0000 complete [ 2325.473656] LustreError: 66647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.499302] LustreError: 66647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2325.594977] LustreError: 64787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102762 with bad export cookie 4985449729219146266 [ 2325.597678] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2325.617119] LustreError: 64787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2325.954665] Lustre: server umount lustre-MDT0001 complete [ 2332.861493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2342.965030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.951168] Lustre: 63720:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102767/real 1786102767] req@000000001613f936 x1872864260201216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786102783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2345.991263] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.119178] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x452fe01d5bacb77d [ 2350.141434] Lustre: MGC192.168.202.150@tcp: Connection restored to 0@lo (at 0@lo) [ 2350.584377] LustreError: 66647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.644662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.680428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2356.711068] Lustre: 63721:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786102777/real 1786102777] req@0000000055564897 x1872864260201600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786102793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2356.747673] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2356.751723] Lustre: 63721:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2370.572773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.527322] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2377.210779] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2387.075088] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2393.445666] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2401.257171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.259985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.267158] Lustre: Skipped 1 previous similar message [ 2401.305193] Lustre: Skipped 4 previous similar messages [ 2406.384424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.399445] Lustre: Skipped 2 previous similar messages [ 2407.074385] Lustre: server umount lustre-MDT0000 complete [ 2411.489675] LustreError: 72695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.527231] LustreError: 72695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2411.647122] LustreError: 64788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102848 with bad export cookie 4985449729219147645 [ 2411.656960] LustreError: 64788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2411.659580] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.186675] Lustre: server umount lustre-MDT0001 complete [ 2420.621741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2424.885092] Lustre: DEBUG MARKER: files compared the same [ 2436.659480] Lustre: server umount lustre-OST0000 complete [ 2455.994946] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2459.405338] Key type lgssc unregistered [ 2459.731601] LNet: 74537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.750523] LNetError: 74537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2460.783650] LNet: Removed LNI 192.168.202.150@tcp [ 2462.018846] Key type .llcrypt unregistered [ 2462.020337] Key type ._llcrypt unregistered [ 2485.758819] Key type ._llcrypt registered [ 2485.762413] Key type .llcrypt registered [ 2485.889633] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2487.152708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.196089] alg: No test for adler32 (adler32-zlib) [ 2488.378601] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2488.680963] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2490.351277] Key type lgssc registered [ 2492.011680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.995106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2501.037205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.646978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.495965] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug +malloc all [ 2512.876531] LustreError: 75955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.907775] LustreError: 75955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2517.865089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.991810] LustreError: 75956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.291991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2524.187287] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug +malloc all [ 2532.960701] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2539.296694] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2546.100511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2546.441673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.459824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2553.774907] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug +malloc all [ 2562.416654] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2573.282181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2573.291564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.303493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.304986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.308001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.313441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.238528] Lustre: server umount lustre-OST0000 complete [ 2584.544556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.568259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.574592] Lustre: Skipped 1 previous similar message [ 2589.664136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.679077] Lustre: Skipped 1 previous similar message [ 2594.784455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.791325] Lustre: Skipped 1 previous similar message [ 2596.319136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.547597] Lustre: server umount lustre-MDT0000 complete [ 2599.904471] LustreError: 76684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.923217] LustreError: 76684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2600.713212] LustreError: 75935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103037 with bad export cookie 14547010365794386504 [ 2600.727521] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.730422] LustreError: 75935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2601.009634] Lustre: server umount lustre-MDT0001 complete [ 2614.386592] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2618.050801] Key type lgssc unregistered [ 2618.423767] LNet: 79492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2618.439589] LNetError: 79492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2618.460367] LNet: Removed LNI 192.168.202.150@tcp [ 2619.590337] Key type .llcrypt unregistered [ 2619.593895] Key type ._llcrypt unregistered [ 2649.362681] Key type ._llcrypt registered [ 2649.365189] Key type .llcrypt registered [ 2649.467491] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2650.922191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2651.211663] alg: No test for adler32 (adler32-zlib) [ 2652.444526] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2652.739921] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2654.471171] Key type lgssc registered [ 2655.858262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.690547] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2664.724522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2680.287723] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2685.408034] LustreError: 80887:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 2712.097235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.033353] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2727.695606] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.491258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.642858] Lustre: *** cfs_fail_loc=706, val=0*** [ 2737.000367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.120344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2742.252747] LustreError: 80913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.274054] LustreError: 80913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2742.616580] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2746.409956] LustreError: 80912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.437520] LustreError: 80912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2751.471422] LustreError: 80913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.491312] LustreError: 80913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2752.066260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.640207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.890375] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2766.673443] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2772.320703] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2778.086163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.101304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.115896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.124260] Lustre: Skipped 1 previous similar message [ 2778.160443] Lustre: Skipped 1 previous similar message [ 2780.899522] Lustre: server umount lustre-OST0000 complete [ 2788.321306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2788.339327] LustreError: Skipped 1 previous similar message [ 2788.357306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.372065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.164619] Lustre: server umount lustre-MDT0000 complete [ 2793.952500] LustreError: 81977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.969308] LustreError: 81977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2795.475575] LustreError: 80893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103232 with bad export cookie 5802112237715858909 [ 2795.478400] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2795.490149] LustreError: 80893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2795.871773] Lustre: server umount lustre-MDT0001 complete [ 2806.889985] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2809.668885] Key type lgssc unregistered [ 2810.022926] LNet: 84507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.038988] LNetError: 84507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2810.077048] LNet: Removed LNI 192.168.202.150@tcp [ 2811.146742] Key type .llcrypt unregistered [ 2811.152085] Key type ._llcrypt unregistered [ 2842.453429] Key type ._llcrypt registered [ 2842.454844] Key type .llcrypt registered [ 2842.505271] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2843.863129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2843.899524] alg: No test for adler32 (adler32-zlib) [ 2845.481944] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 2845.629876] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2847.433134] Key type lgssc registered [ 2849.472224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2859.858411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2859.872222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.507271] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2876.168810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.748807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2882.029735] LustreError: 86612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.044325] LustreError: 86612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2882.323503] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2886.113364] LustreError: 86610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.132453] LustreError: 86610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2891.251979] LustreError: 86615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.279376] LustreError: 86615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2891.971798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.472159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.888727] LustreError: 86612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.903374] LustreError: 86612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2900.096504] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2906.083241] LustreError: 86612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.120168] LustreError: 86612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2910.623718] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.645107] Lustre: MGS: Client f8c474cf-860b-4a19-b042-f1117fb38d69 (at 0@lo) reconnecting [ 2910.656071] LustreError: 86626:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2910.918311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2911.008899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2916.544636] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2932.708770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.717641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.738732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2936.810563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.819063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2936.843892] Lustre: Skipped 1 previous similar message [ 2939.097834] Lustre: server umount lustre-OST0000 complete [ 2947.040944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.041524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2947.066878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.076397] Lustre: Skipped 1 previous similar message [ 2949.569177] Lustre: server umount lustre-MDT0000 complete [ 2952.160470] LustreError: 88156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.197568] LustreError: 88156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2955.506448] Lustre: server umount lustre-MDT0001 complete [ 2960.292141] Lustre: server umount MGS complete [ 2975.168932] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:50:10 (1786103410) [ 3006.533284] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 3016.460744] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3052.350766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.260853] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3067.057233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.266322] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3067.308804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3067.422381] Lustre: lustre-MDT0001: new disk, initializing [ 3067.518966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.554716] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3067.567229] Lustre: 93808:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3067.579376] LustreError: 93808:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3072.603155] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3084.075596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3084.313521] Lustre: lustre-OST0000: new disk, initializing [ 3084.316756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3084.320839] Lustre: 94573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3084.402896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.376614] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3091.552110] Lustre: 94930:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000014e912c3 x1872865035069568/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786103544 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3099.130650] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.150797] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.176757] Lustre: MGS: Client b79669cc-88ff-4697-b09f-618b6ea5fba3 (at 0@lo) reconnecting [ 3105.220488] Lustre: 93077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3105.234762] LustreError: 93794:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3105.267739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3105.361421] Lustre: lustre-MDT0000: new disk, initializing [ 3105.529651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.550226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3105.671123] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3106.600577] Lustre: 95641:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000875a38d8 x1872865035076096/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786103559 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3110.513386] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3111.908764] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3115.757223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3115.776293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3115.845592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3141.611948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.618633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.631704] Lustre: Skipped 1 previous similar message [ 3141.636331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.672448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.702224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3145.710665] Lustre: Skipped 2 previous similar messages [ 3147.085716] Lustre: server umount lustre-OST0000 complete [ 3155.941487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.944356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.950246] Lustre: Skipped 1 previous similar message [ 3155.958854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.181220] Lustre: server umount lustre-MDT0000 complete [ 3175.391348] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3175.708653] Lustre: server umount lustre-MDT0001 complete [ 3179.762725] Lustre: server umount MGS complete [ 3190.393383] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:53:46 (1786103626) [ 3207.075375] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3210.022926] Key type lgssc unregistered [ 3210.301109] LNet: 98146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3210.316249] LNetError: 98146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3210.333488] LNet: Removed LNI 192.168.202.150@tcp [ 3211.180247] Key type .llcrypt unregistered [ 3211.183846] Key type ._llcrypt unregistered [ 3227.574688] Key type ._llcrypt registered [ 3227.577618] Key type .llcrypt registered [ 3227.649810] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3229.191480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3229.234954] alg: No test for adler32 (adler32-zlib) [ 3230.458084] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 3230.720526] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3232.391202] Key type lgssc registered [ 3233.460672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3241.098429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3241.107958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.640731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.657993] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3252.205815] LustreError: 99366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.221551] LustreError: 99366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3255.961602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.372600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3260.220441] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3266.706376] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3271.759651] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3277.132304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.400975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.630464] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3287.668985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3289.422580] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3298.275097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.281739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.296915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.326406] Lustre: Skipped 1 previous similar message [ 3303.397178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3303.404941] Lustre: Skipped 1 previous similar message [ 3303.717684] Lustre: server umount lustre-OST0000 complete [ 3307.489916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3307.502667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.514243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.521276] Lustre: Skipped 1 previous similar message [ 3308.516140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.525618] Lustre: server umount lustre-MDT0000 complete [ 3313.648056] LustreError: 99367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.682988] LustreError: 99367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3317.129161] LustreError: 99348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103754 with bad export cookie 18346410800830691269 [ 3317.142442] LustreError: 99348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3317.143969] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3317.464928] Lustre: server umount lustre-MDT0001 complete [ 3326.657387] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3329.152075] Key type lgssc unregistered [ 3329.412625] LNet: 102900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.416834] LNetError: 102900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.473649] LNet: Removed LNI 192.168.202.150@tcp [ 3331.204224] Key type .llcrypt unregistered [ 3331.206051] Key type ._llcrypt unregistered [ 3343.899771] Key type ._llcrypt registered [ 3343.901551] Key type .llcrypt registered [ 3343.990694] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3344.727107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3344.793306] alg: No test for adler32 (adler32-zlib) [ 3345.757712] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 3345.930330] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3347.567318] Key type lgssc registered [ 3348.429163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3354.235305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3354.237564] LustreError: 103966:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3354.243955] Lustre: Skipped 1 previous similar message [ 3354.243975] LustreError: 103963:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3354.265220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.268201] LustreError: 103966:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3354.291529] LustreError: 103966:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3354.297617] LustreError: 103966:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3355.766756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.094625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.443620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.208865] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3373.610698] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing lsmod [ 3376.143570] LustreError: 105173:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3376.145494] LustreError: 105172:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3376.161701] LustreError: 105173:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3376.171404] LustreError: 105173:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3376.179552] LustreError: 105173:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3376.192826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.573475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.730816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.735169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.745039] Lustre: Skipped 1 previous similar message [ 3381.758484] Lustre: Skipped 2 previous similar messages [ 3385.508475] Lustre: server umount lustre-MDT0000 complete [ 3386.848810] LustreError: 104544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.865443] LustreError: 104544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3388.906418] LustreError: 105358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.921282] LustreError: 105358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3389.061325] LustreError: 103980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103826 with bad export cookie 18184873607942943751 [ 3389.066046] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.071032] LustreError: 103980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3391.007497] LustreError: 104435:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b168966b x1872865560382720/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3391.223162] Lustre: server umount lustre-MDT0001 complete [ 3405.410764] Lustre: server umount lustre-OST0000 complete [ 3413.350425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.722686] LustreError: 106237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.785750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.849939] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3419.109808] LustreError: 106238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.204389] LustreError: 106238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.837102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.179505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3430.416919] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3437.323555] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3441.894610] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3448.027544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.400119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.194796] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3458.683738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3461.111542] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3467.232243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3467.245290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3467.259065] Lustre: Skipped 1 previous similar message [ 3467.262886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.285628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.288409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.293484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.113743] Lustre: server umount lustre-OST0000 complete [ 3477.479609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3477.491945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.519378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3477.522529] Lustre: Skipped 1 previous similar message [ 3484.641890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.658547] Lustre: Skipped 3 previous similar messages [ 3491.295210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3491.508333] Lustre: server umount lustre-MDT0000 complete [ 3494.885763] LustreError: 107285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.908723] LustreError: 107285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3495.237572] LustreError: 106218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786103932 with bad export cookie 18184873607942944437 [ 3495.241796] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.244819] LustreError: 106218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3495.671404] Lustre: server umount lustre-MDT0001 complete [ 3507.084292] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3509.855217] Key type lgssc unregistered [ 3510.159433] LNet: 109772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3510.164342] LNetError: 109772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3510.179077] LNet: Removed LNI 192.168.202.150@tcp [ 3510.965233] Key type .llcrypt unregistered [ 3510.966504] Key type ._llcrypt unregistered [ 3534.838415] Key type ._llcrypt registered [ 3534.840892] Key type .llcrypt registered [ 3535.000165] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3536.439218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3536.481554] alg: No test for adler32 (adler32-zlib) [ 3537.568559] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 3537.831777] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3539.511637] Key type lgssc registered [ 3540.592886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3547.286958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3547.297889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.682508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3554.234254] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3558.884556] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.919699] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3564.002100] LustreError: 111191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.917097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.200237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3579.487512] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3591.493981] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3598.540791] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3605.781783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.245126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.571936] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3618.557119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3622.499653] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3627.719298] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3632.278952] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3639.273408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3639.276360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.288172] LustreError: Skipped 1 previous similar message [ 3639.321161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3639.357202] Lustre: Skipped 1 previous similar message [ 3642.248729] Lustre: server umount lustre-OST0000 complete [ 3649.504844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.513566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.516591] Lustre: Skipped 2 previous similar messages [ 3649.530177] Lustre: Skipped 1 previous similar message [ 3652.585069] Lustre: server umount lustre-MDT0000 complete [ 3654.630037] LustreError: 111191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.642058] LustreError: 111191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3655.827423] LustreError: 111171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104092 with bad export cookie 14047625925108889139 [ 3655.831792] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3655.834169] LustreError: 111171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3656.179387] Lustre: server umount lustre-MDT0001 complete [ 3667.209929] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3669.533525] Key type lgssc unregistered [ 3669.847456] LNet: 114845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3669.863904] LNetError: 114845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3669.893624] LNet: Removed LNI 192.168.202.150@tcp [ 3670.879616] Key type .llcrypt unregistered [ 3670.882739] Key type ._llcrypt unregistered [ 3687.455853] Key type ._llcrypt registered [ 3687.459504] Key type .llcrypt registered [ 3687.562420] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3688.873672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3688.909990] alg: No test for adler32 (adler32-zlib) [ 3690.044109] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 3690.281456] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3692.031165] Key type lgssc registered [ 3692.884716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3703.160738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3703.174479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.468296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.761282] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3714.016325] LustreError: 116069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3714.025848] LustreError: 116069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3719.136602] LustreError: 116068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3719.448818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.090996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3725.420778] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3733.131790] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3738.541677] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3746.663468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.109634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.184418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3753.582503] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3761.637654] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3773.932650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3773.939387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.951465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.978140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.995745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3776.002219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.098992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.115127] Lustre: Skipped 2 previous similar messages [ 3786.230135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.231130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3787.433122] Lustre: server umount lustre-OST0000 complete [ 3796.451448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.457242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.472060] Lustre: Skipped 2 previous similar messages [ 3798.253397] Lustre: server umount lustre-MDT0000 complete [ 3801.579063] LustreError: 116068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.603870] LustreError: 116068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3802.501348] LustreError: 116048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104239 with bad export cookie 3799741228866143024 [ 3802.514900] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.049513] Lustre: server umount lustre-MDT0001 complete [ 3814.593344] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3817.600887] Key type lgssc unregistered [ 3817.913638] LNet: 119605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3817.926240] LNetError: 119605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3817.957417] LNet: Removed LNI 192.168.202.150@tcp [ 3818.821596] Key type .llcrypt unregistered [ 3818.823572] Key type ._llcrypt unregistered [ 3845.226634] Key type ._llcrypt registered [ 3845.228974] Key type .llcrypt registered [ 3845.373468] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3847.118578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3847.173351] alg: No test for adler32 (adler32-zlib) [ 3848.308645] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 3848.636109] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3850.311159] Key type lgssc registered [ 3851.642470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3859.585022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3859.603376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.977860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.750886] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3871.207924] LustreError: 121028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.241928] LustreError: 121028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3874.069779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.359080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3878.630306] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3887.101699] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3893.781452] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3901.143771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.647508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.837140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3908.643359] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3917.902554] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3924.947707] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3928.705533] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3928.711509] Lustre: Skipped 2 previous similar messages [ 3932.570282] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3932.572940] Lustre: Skipped 2 previous similar messages [ 3932.796219] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3932.824671] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3939.672528] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.50@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3939.682562] Lustre: Skipped 2 previous similar messages [ 3942.748163] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.50@tcp 192.168.0.[2,10]@tcp [ 3948.199782] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.50/24@tcp in log lustre-MDT0000 [ 3948.205212] Lustre: Skipped 2 previous similar messages [ 3952.726485] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.50/24@tcp [ 3952.738594] Lustre: Skipped 1 previous similar message [ 3960.292077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.303549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.319268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3960.327963] Lustre: Skipped 1 previous similar message [ 3965.410289] LustreError: 122852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.431626] LustreError: 122852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3965.513900] Lustre: server umount lustre-OST0000 complete [ 3970.529915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.538761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.554202] Lustre: Skipped 1 previous similar message [ 3970.569303] Lustre: Skipped 2 previous similar messages [ 3975.649515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.659632] Lustre: Skipped 1 previous similar message [ 3980.769147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.778181] Lustre: Skipped 1 previous similar message [ 3984.351319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3984.704451] Lustre: server umount lustre-MDT0000 complete [ 3985.888247] LustreError: 122038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.616276] LustreError: 124262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104425 with bad export cookie 16363965059311045149 [ 3988.617688] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.626082] LustreError: 124262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3989.140747] Lustre: server umount lustre-MDT0001 complete [ 3998.873193] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 4001.696736] Key type lgssc unregistered [ 4002.092790] LNet: 125727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4002.103473] LNetError: 125727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.173812] LNet: Removed LNI 192.168.202.150@tcp [ 4004.074946] Key type .llcrypt unregistered [ 4004.076146] Key type ._llcrypt unregistered [ 4033.918043] Key type ._llcrypt registered [ 4033.920128] Key type .llcrypt registered [ 4034.068159] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4035.465281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.508431] alg: No test for adler32 (adler32-zlib) [ 4036.613262] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 4036.934868] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 4038.647570] Key type lgssc registered [ 4039.658619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4046.715624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4046.733165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.152027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.155678] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4048.179956] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.50/24@tcp [ 4052.555365] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4058.592902] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.639236] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4061.746675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.082627] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4062.123412] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.50/24@tcp [ 4062.226429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.767980] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4074.377208] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4079.231752] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4086.290855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4086.618878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.576469] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4093.820718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4101.388259] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4113.377084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.387556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.407040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.403596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.404860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.425993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.676067] Lustre: server umount lustre-OST0000 complete [ 4123.617403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4123.629162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.644056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4123.648676] Lustre: Skipped 1 previous similar message [ 4129.431697] Lustre: server umount lustre-MDT0000 complete [ 4129.764520] LustreError: 127284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.778026] LustreError: 127284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4133.650755] LustreError: 127269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104570 with bad export cookie 16480173572487182290 [ 4133.664371] LustreError: 127269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4133.675657] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.968606] Lustre: server umount lustre-MDT0001 complete [ 4145.942282] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 4149.054226] Key type lgssc unregistered [ 4149.410841] LNet: 130881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4149.420969] LNetError: 130881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4149.442697] LNet: Removed LNI 192.168.202.150@tcp [ 4150.460173] Key type .llcrypt unregistered [ 4150.461903] Key type ._llcrypt unregistered [ 4177.200280] Key type ._llcrypt registered [ 4177.203499] Key type .llcrypt registered [ 4177.380440] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 4192.752325] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4194.144258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4194.210480] alg: No test for adler32 (adler32-zlib) [ 4195.256652] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 4195.540649] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 4197.247544] Key type lgssc registered [ 4198.583939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4231.897130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4231.938609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.207819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4233.245385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4233.308591] Lustre: lustre-MDT0000: new disk, initializing [ 4233.431193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.457160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4237.988738] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4249.209735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.409435] Lustre: 134283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4249.554530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4249.574336] Lustre: Skipped 1 previous similar message [ 4249.663749] Lustre: lustre-MDT0001: new disk, initializing [ 4249.798627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4249.874210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4249.894578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4255.015851] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4264.431720] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4269.714656] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4276.900556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4277.170955] Lustre: lustre-OST0000: new disk, initializing [ 4277.175343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4277.184545] Lustre: 136251:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4277.306776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.985831] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4287.037689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4287.045694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4287.202051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4291.777558] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.250762] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4297.463927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4302.508992] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4302.685888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4307.937511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.942176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.952478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.958763] Lustre: Skipped 1 previous similar message [ 4310.832813] Lustre: server umount lustre-OST0000 complete [ 4316.640111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.656624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.667387] Lustre: Skipped 1 previous similar message [ 4316.676430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.176358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.177603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.198555] Lustre: Skipped 1 previous similar message [ 4321.473125] Lustre: server umount lustre-MDT0000 complete [ 4325.374342] LustreError: 135078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104762 with bad export cookie 17334226788498629705 [ 4325.375872] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.383784] LustreError: 135078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4325.725769] Lustre: server umount lustre-MDT0001 complete [ 4335.337875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.963376] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.979119] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4336.041234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.229270] LustreError: 138468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.261489] LustreError: 138468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4341.262818] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4346.339095] LustreError: 138492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.228934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.459700] LustreError: 138468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.740206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4356.777561] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4365.236262] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4373.278066] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4381.625563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.106128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.994544] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4399.025991] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4404.178057] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4407.631988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4411.322426] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4411.534713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4420.578271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.718920] Lustre: lustre-OST0001: new disk, initializing [ 4420.722484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4420.729635] Lustre: 141859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4420.814569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.818348] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4427.341192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4427.352612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4427.499700] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4435.982264] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4436.209536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4441.314612] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4441.556890] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4447.488411] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a78927b9800.ost_server_uuid 50 [ 4449.149958] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a78927b9800.ost_server_uuid in IDLE state after 0 sec [ 4458.486099] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4458.493823] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.536940] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4458.542050] Lustre: Skipped 1 previous similar message [ 4462.676820] Lustre: server umount lustre-OST0001 complete [ 4463.588506] LustreError: 140290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.620828] LustreError: 140290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4468.194682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.196949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.210295] LustreError: Skipped 2 previous similar messages [ 4468.231639] Lustre: Skipped 1 previous similar message [ 4468.232456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4472.845306] Lustre: server umount lustre-OST0000 complete [ 4478.960560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.982774] Lustre: Skipped 1 previous similar message [ 4479.000632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.009233] Lustre: Skipped 2 previous similar messages [ 4483.636707] Lustre: server umount lustre-MDT0000 complete [ 4484.065576] LustreError: 138464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.081434] LustreError: 138464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4487.771924] LustreError: 138448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786104924 with bad export cookie 17334226788498630587 [ 4487.778029] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.792037] LustreError: 138448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4488.177645] Lustre: server umount lustre-MDT0001 complete [ 4510.524366] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 4513.898047] Key type lgssc unregistered [ 4514.233339] LNet: 144868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4514.238396] LNetError: 144868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4514.251306] LNet: Removed LNI 192.168.202.150@tcp [ 4515.111095] Key type .llcrypt unregistered [ 4515.112890] Key type ._llcrypt unregistered [ 4555.102624] Key type ._llcrypt registered [ 4555.104104] Key type .llcrypt registered [ 4555.232798] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 4568.545575] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4569.610443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4569.629792] alg: No test for adler32 (adler32-zlib) [ 4570.746601] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 4571.005330] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 4572.680183] Key type lgssc registered [ 4573.646619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4610.559505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4610.600535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.851213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4611.876725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4611.929962] Lustre: lustre-MDT0000: new disk, initializing [ 4611.990480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.006607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.348827] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4628.531399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.703605] Lustre: 149004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4628.814244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4628.818219] Lustre: Skipped 1 previous similar message [ 4628.883426] Lustre: lustre-MDT0001: new disk, initializing [ 4628.957021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4629.029253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4629.036628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4635.201490] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4646.008622] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4653.129555] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4661.806561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.304482] Lustre: lustre-OST0000: new disk, initializing [ 4662.308523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4662.321995] Lustre: 150973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4662.485239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.490389] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4671.558725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4671.562875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4671.670105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4680.979741] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4686.827191] Lustre: Failing over lustre-OST0000 [ 4687.317702] Lustre: server umount lustre-OST0000 complete [ 4703.185853] Lustre: 147228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786105124/real 1786105124] req@00000000b5d8e282 x1872866844903296/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786105140 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4703.218283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.223372] Lustre: 147227:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786105124/real 1786105124] req@00000000125953ff x1872866844904192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786105140 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4704.243961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.079846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4707.349197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.379612] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4708.319228] Lustre: 147229:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786105129/real 1786105129] req@0000000025828ab5 x1872866844905472/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786105145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4708.365018] Lustre: 147229:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4712.941754] Lustre: 147230:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786105134/real 1786105134] req@00000000a7e3f6b3 x1872866844906496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786105150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4713.526727] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4714.674770] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4714.675756] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4714.823987] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4718.957317] Lustre: Failing over lustre-MDT0000 [ 4719.294552] Lustre: server umount lustre-MDT0000 complete [ 4719.968674] Lustre: 147230:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786105139/real 1786105139] req@000000000055857a x1872866844907648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786105155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4719.991393] Lustre: 147230:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4721.128316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4721.139531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.692634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.711159] Lustre: Skipped 1 previous similar message [ 4724.858261] LustreError: 151588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.873659] LustreError: 151588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4728.801548] LustreError: 149888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.816219] LustreError: 149888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4729.993578] LustreError: 151588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.921698] LustreError: 149014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.950401] LustreError: 149014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4739.041194] LustreError: 149888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.051011] LustreError: 149888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4739.871095] Lustre: 147228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786105160/real 1786105160] req@0000000051211fac x1872866844921344/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786105176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4739.906144] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4741.805883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.309702] LustreError: 149888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.311360] LustreError: 147226:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000157a73ee x1872866844926720/t0(0) o250->MGC192.168.202.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4750.333636] LustreError: 149888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4750.467552] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (not set up) [ 4750.642360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.695422] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4755.132356] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4755.530278] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4755.963184] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4755.965770] Lustre: Skipped 1 previous similar message [ 4756.004043] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4756.062690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4766.196463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4766.207911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.231235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.261708] Lustre: Skipped 1 previous similar message [ 4766.279590] Lustre: Skipped 1 previous similar message [ 4771.054620] Lustre: server umount lustre-OST0000 complete [ 4776.417611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4776.424348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.479151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4776.498943] Lustre: Skipped 1 previous similar message [ 4781.538513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4781.553603] Lustre: Skipped 1 previous similar message [ 4782.582038] Lustre: server umount lustre-MDT0000 complete [ 4786.655768] LustreError: 148996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786105223 with bad export cookie 18284843286240217005 [ 4786.659842] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.667113] LustreError: 148996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4787.065271] Lustre: server umount lustre-MDT0001 complete [ 4802.083137] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 4805.513430] Key type lgssc unregistered [ 4805.831696] LNet: 154635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4805.837731] LNetError: 154635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4805.867092] LNet: Removed LNI 192.168.202.150@tcp [ 4806.890373] Key type .llcrypt unregistered [ 4806.892701] Key type ._llcrypt unregistered [ 4844.771476] Key type ._llcrypt registered [ 4844.773665] Key type .llcrypt registered [ 4844.864198] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4846.145800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4846.250070] alg: No test for adler32 (adler32-zlib) [ 4847.343691] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 4847.635764] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 4849.375719] Key type lgssc registered [ 4850.819600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4858.567706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4858.594087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.948689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.961440] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4870.119952] LustreError: 156159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.134648] LustreError: 156159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4872.632474] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4875.247089] LustreError: 156160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.426186] LustreError: 156188:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4876.438501] LustreError: 156188:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4876.453187] LustreError: 156188:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4876.922054] Lustre: server umount lustre-MDT0000 complete [ 4892.515369] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 4905.126385] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4945.930448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.198977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4946.218671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4946.288805] Lustre: lustre-MDT0000: new disk, initializing [ 4946.373546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4946.394603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4952.148415] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4964.261391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.497052] Lustre: 159494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4964.637924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4964.645503] Lustre: Skipped 1 previous similar message [ 4964.793280] Lustre: lustre-MDT0001: new disk, initializing [ 4964.907572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.944814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4964.961702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4970.173332] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4981.024918] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4987.048650] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4994.248500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.533709] Lustre: lustre-OST0000: new disk, initializing [ 4994.537039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4994.545023] Lustre: 161465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4994.659200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.864506] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5004.363645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5004.377240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5004.466419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5010.934202] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5015.854658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5017.816734] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:24:14 (1786105454) [ 5296.616533] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.625116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5296.625708] Lustre: Skipped 1 previous similar message [ 5296.639369] Lustre: Skipped 2 previous similar messages [ 5300.932283] Lustre: server umount lustre-MDT0000 complete [ 5301.728674] LustreError: 160771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.753487] LustreError: 160771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5304.788711] LustreError: 159489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786105741 with bad export cookie 3018919650588410419 [ 5304.788937] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5304.794179] LustreError: 159489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5306.848550] LustreError: 159503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.849720] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5306.886422] Lustre: Skipped 1 previous similar message [ 5306.889754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5309.858207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5313.955132] LustreError: 160771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5313.955125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5313.985583] LustreError: 160771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5319.140077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5319.142972] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5319.526140] Lustre: server umount lustre-MDT0001 complete [ 5330.835360] Lustre: server umount lustre-OST0000 complete [ 5341.846241] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 5351.097952] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5388.085140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.279172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5388.302058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5388.360293] Lustre: lustre-MDT0000: new disk, initializing [ 5388.426344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.440033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5392.682570] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5403.987681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.118618] Lustre: 166133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5404.163770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5404.166787] Lustre: Skipped 1 previous similar message [ 5404.242783] Lustre: lustre-MDT0001: new disk, initializing [ 5404.364666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.400160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5404.426541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5409.195111] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5419.332130] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5424.815792] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5432.674959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5432.929253] Lustre: lustre-OST0000: new disk, initializing [ 5432.933917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5432.938877] Lustre: 168100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5433.007447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.129732] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5440.677956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5440.694410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5440.791991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5452.155615] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5457.950595] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5458.120748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5465.867709] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5466.267721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5471.715332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5471.723070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.733955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5471.743181] Lustre: Skipped 1 previous similar message [ 5474.798306] Lustre: server umount lustre-OST0000 complete [ 5481.447221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5481.457394] LustreError: Skipped 1 previous similar message [ 5481.462713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.478108] Lustre: Skipped 1 previous similar message [ 5485.611542] Lustre: server umount lustre-MDT0000 complete [ 5487.076227] LustreError: 166139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.108444] LustreError: 166139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5489.762407] LustreError: 166126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786105926 with bad export cookie 3018919650588569249 [ 5489.765471] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.771662] LustreError: 166126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5490.276918] Lustre: server umount lustre-MDT0001 complete [ 5500.115645] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:32:16 (1786105936) [ 5517.171623] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 5527.979755] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5567.879793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.122467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5568.145835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5568.228881] Lustre: lustre-MDT0000: new disk, initializing [ 5568.304966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.325079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5573.540515] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5584.808565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.033856] Lustre: 172863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5585.176732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5585.180408] Lustre: Skipped 2 previous similar messages [ 5585.406692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5591.058739] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5601.408866] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5609.786728] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5621.477289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.152882] Lustre: lustre-OST0000: new disk, initializing [ 5622.159956] Lustre: Skipped 1 previous similar message [ 5622.169981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5622.175855] Lustre: Skipped 2 previous similar messages [ 5622.187186] Lustre: 174843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5630.631196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5630.639551] Lustre: Skipped 1 previous similar message [ 5630.642174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5630.732367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5631.309257] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5641.484458] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5651.425658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.432917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.439251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.440674] Lustre: Skipped 2 previous similar messages [ 5654.975899] Lustre: server umount lustre-OST0000 complete [ 5661.665573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.690049] Lustre: Skipped 1 previous similar message [ 5666.375050] Lustre: server umount lustre-MDT0000 complete [ 5666.784642] LustreError: 172875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.827712] LustreError: 172875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5671.391257] LustreError: 172856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106108 with bad export cookie 3018919650588570131 [ 5671.422754] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.992146] Lustre: server umount lustre-MDT0001 complete [ 5682.499439] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:35:18 (1786106118) [ 5700.671647] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 5711.933479] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5751.786464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5752.059746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5752.116528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5752.202184] Lustre: lustre-MDT0000: new disk, initializing [ 5752.266629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.273211] Lustre: Skipped 2 previous similar messages [ 5752.294599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5757.029939] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5768.340267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.486881] Lustre: 179180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5768.698424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5773.690932] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5784.843291] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5791.256212] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5798.668348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.063081] Lustre: 181159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5805.151345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5805.322283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5805.521848] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5815.326806] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5826.030381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5826.042824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.046760] LustreError: Skipped 1 previous similar message [ 5826.047284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.047287] Lustre: Skipped 3 previous similar messages [ 5826.129350] Lustre: Skipped 1 previous similar message [ 5831.165232] Lustre: server umount lustre-OST0000 complete [ 5842.270588] Lustre: server umount lustre-MDT0000 complete [ 5846.515783] LustreError: 179194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.549797] LustreError: 179194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5849.759714] LustreError: 179173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106286 with bad export cookie 3018919650588571209 [ 5849.771734] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5849.780887] LustreError: 179173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5850.733168] Lustre: server umount lustre-MDT0001 complete [ 5864.592278] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 5867.122774] Key type lgssc unregistered [ 5867.425861] LNet: 183020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5867.429728] LNetError: 183020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5867.452622] LNet: Removed LNI 192.168.202.150@tcp [ 5868.262301] Key type .llcrypt unregistered [ 5868.264521] Key type ._llcrypt unregistered [ 5900.242213] Key type ._llcrypt registered [ 5900.245261] Key type .llcrypt registered [ 5900.418282] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5902.082872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5902.109079] alg: No test for adler32 (adler32-zlib) [ 5903.390142] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 5903.759422] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 5905.479847] Key type lgssc registered [ 5907.647881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5919.686284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5919.718868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.424857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.368600] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5930.992845] LustreError: 184437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.008541] LustreError: 184437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5936.100820] LustreError: 184438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.148737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5939.693775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.780988] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5954.165647] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5960.355466] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5967.866473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.169544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.666527] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5983.923202] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5992.837196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6013.423447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.425151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6013.431180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6017.230511] Lustre: server umount lustre-OST0000 complete [ 6023.654086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.665492] Lustre: Skipped 1 previous similar message [ 6023.669852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.676938] Lustre: Skipped 2 previous similar messages [ 6028.131970] Lustre: server umount lustre-MDT0000 complete [ 6028.774625] LustreError: 184438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6028.792130] LustreError: 184438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6031.743396] LustreError: 184417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106468 with bad export cookie 9304919802323866433 [ 6031.751450] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6031.756034] LustreError: 184417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6032.373609] Lustre: server umount lustre-MDT0001 complete [ 6044.140096] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6047.479616] Key type lgssc unregistered [ 6047.672786] LNet: 188004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6047.679238] LNetError: 188004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6047.696386] LNet: Removed LNI 192.168.202.150@tcp [ 6048.673094] Key type .llcrypt unregistered [ 6048.674673] Key type ._llcrypt unregistered [ 6076.996940] Key type ._llcrypt registered [ 6076.999622] Key type .llcrypt registered [ 6077.116079] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6078.956790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6078.970232] alg: No test for adler32 (adler32-zlib) [ 6079.911967] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 6080.097607] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6081.767253] Key type lgssc registered [ 6082.798785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6095.700848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6095.728427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.235283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.644780] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6108.658291] LustreError: 189421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.692521] LustreError: 189421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6113.768873] LustreError: 189422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.887707] LustreError: 189421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.973780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.892348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.305119] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6139.035928] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6145.837829] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6153.658769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6154.120494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6161.372746] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6168.518098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6169.902437] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6175.888173] Lustre: lustre-OST0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 6176.738262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6176.740976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6176.755966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6178.788241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6178.790325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6178.795744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.750303] Lustre: server umount lustre-OST0000 complete [ 6188.828734] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a789c8aa000.ost_server_uuid 50 [ 6190.838771] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6208.998249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6209.005062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.034813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.044805] Lustre: Skipped 1 previous similar message [ 6210.771529] Lustre: server umount lustre-MDT0000 complete [ 6213.985128] LustreError: 189401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106651 with bad export cookie 10907848212441331602 [ 6213.989810] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6213.994084] LustreError: 189401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6214.233668] Lustre: server umount lustre-MDT0001 complete [ 6222.357629] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:44:18 (1786106658) [ 6229.150161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.461380] LustreError: 193337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6229.543920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6233.676115] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6234.608338] LustreError: 193338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.214141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.756372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6248.221580] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6258.171972] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6265.370846] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6273.627207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.149715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.231375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6284.080158] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6299.182882] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6309.796911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6309.989145] Lustre: lustre-OST0001: new disk, initializing [ 6309.994672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6310.005712] Lustre: 196182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6310.111778] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6316.994863] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6319.841359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6319.864791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6319.994057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6328.186120] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.936769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6333.940053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.981388] Lustre: Skipped 1 previous similar message [ 6333.985305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6333.991758] Lustre: Skipped 2 previous similar messages [ 6338.647843] Lustre: server umount lustre-OST0000 complete [ 6339.208633] LustreError: 195160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.233853] LustreError: 195160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6348.436052] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6348.720124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6353.934832] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6354.289546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6355.939711] LustreError: 195160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.967742] LustreError: 195160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6371.308458] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.316192] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.320323] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6371.320332] Lustre: Skipped 3 previous similar messages [ 6371.323572] Lustre: Skipped 2 previous similar messages [ 6371.342490] LustreError: Skipped 1 previous similar message [ 6375.719813] Lustre: server umount lustre-OST0001 complete [ 6381.025776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6381.031222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.401843] Lustre: server umount lustre-MDT0000 complete [ 6388.767219] LustreError: 193320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786106825 with bad export cookie 10907848212441332673 [ 6388.790135] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6389.183525] Lustre: server umount lustre-MDT0001 complete [ 6421.564332] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:47:38 (1786106858) [ 6429.506114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.829187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6429.891930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6430.202220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.965837] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6448.162481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.400092] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6448.473758] Lustre: 199630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6456.986279] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6468.717918] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6477.425203] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6484.639988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.912958] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6490.770760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6492.269240] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6501.456224] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.076577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.356944] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6513.875242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6513.886322] Lustre: Skipped 2 previous similar messages [ 6524.780382] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6535.751882] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6541.797198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.801787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.808468] Lustre: Skipped 1 previous similar message [ 6541.811835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6541.817335] Lustre: Skipped 3 previous similar messages [ 6542.223771] Lustre: server umount lustre-OST0000 complete [ 6542.310938] LustreError: 201484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.335269] LustreError: 201484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6565.595951] Lustre: server umount lustre-OST0001 complete [ 6575.076592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6575.097503] Lustre: Skipped 8 previous similar messages [ 6583.264053] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6583.725990] Lustre: server umount lustre-MDT0000 complete [ 6589.938690] LustreError: 200387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786107027 with bad export cookie 10907848212441334080 [ 6589.946665] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.958587] LustreError: 200387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6590.652361] Lustre: server umount lustre-MDT0001 complete [ 6621.766624] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:50:56 (1786107056) [ 6649.377431] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 6662.358296] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6701.331114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.657397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6701.698900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6701.810650] Lustre: lustre-MDT0000: new disk, initializing [ 6701.953968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.992843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6706.618492] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6718.002945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.135331] Lustre: 207737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6718.174245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6718.177348] Lustre: Skipped 1 previous similar message [ 6718.226397] Lustre: lustre-MDT0001: new disk, initializing [ 6718.341351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6718.349440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6723.268882] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6734.281405] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6741.230421] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6749.160157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.514846] Lustre: lustre-OST0000: new disk, initializing [ 6749.521975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6749.532145] Lustre: 209706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6757.153624] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6758.003063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6758.024191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6758.197330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6768.342682] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6773.135340] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6773.342781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6779.207915] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6779.441462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6783.970235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6783.979333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.985937] LustreError: Skipped 2 previous similar messages [ 6784.016620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6784.021388] Lustre: Skipped 6 previous similar messages [ 6784.032209] Lustre: Skipped 3 previous similar messages [ 6796.255127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6796.417037] Lustre: server umount lustre-OST0000 complete [ 6815.199667] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6815.376139] Lustre: server umount lustre-MDT0000 complete [ 6819.330761] LustreError: 209705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786107256 with bad export cookie 10907848212441335641 [ 6819.335532] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.341396] LustreError: 209705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6819.626791] Lustre: server umount lustre-MDT0001 complete [ 6829.076962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.427860] LustreError: 211925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.446273] LustreError: 211925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6833.763170] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6843.313481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.229567] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6856.598212] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6862.571532] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.350288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6878.096826] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6888.399423] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6894.306511] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6894.757379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6900.306313] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6900.558911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6903.777626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6903.780306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.796803] LustreError: Skipped 1 previous similar message [ 6903.813970] Lustre: Skipped 2 previous similar messages [ 6914.030685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6914.039785] Lustre: Skipped 15 previous similar messages [ 6917.088625] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6917.207359] Lustre: server umount lustre-OST0000 complete [ 6925.388335] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6931.558852] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6964.112919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.666318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6964.669035] Lustre: Skipped 5 previous similar messages [ 6969.805117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6969.823595] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6969.828162] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6969.868519] Lustre: Skipped 1 previous similar message [ 6974.689621] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6984.732932] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6986.475567] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6997.007244] Lustre: server umount lustre-OST0000 complete [ 7006.983857] Lustre: server umount lustre-MDT0000 complete [ 7010.800208] LustreError: 211920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.822310] LustreError: 211920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7010.918104] LustreError: 211905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786107448 with bad export cookie 10907848212441336523 [ 7010.934883] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7011.203192] Lustre: server umount lustre-MDT0001 complete [ 7019.914054] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:57:36 (1786107456) [ 7026.756744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.381901] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7042.441743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.477688] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7057.439816] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7064.746813] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7071.869540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.232458] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7089.789436] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7095.566391] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7095.798802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7101.208178] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7101.429913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7109.528556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7109.836794] Lustre: lustre-OST0001: new disk, initializing [ 7109.841645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7109.855901] Lustre: 221272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7117.856178] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7118.936278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7118.946844] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7119.150062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7127.699651] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7132.828323] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7133.065857] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7138.200229] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7138.506242] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7144.928905] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7144.941480] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.945260] LustreError: Skipped 3 previous similar messages [ 7144.985132] Lustre: Skipped 6 previous similar messages [ 7146.724158] Lustre: server umount lustre-OST0001 complete [ 7152.847356] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7153.044087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7157.548202] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7157.770390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7188.481152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.093274] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7194.123386] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7194.124298] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7198.040360] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7210.381760] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7212.782843] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7219.702480] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7219.710966] Lustre: Skipped 8 previous similar messages [ 7221.930472] Lustre: server umount lustre-OST0001 complete [ 7246.271822] Lustre: server umount lustre-OST0000 complete [ 7257.699845] Lustre: server umount lustre-MDT0000 complete [ 7262.063461] LustreError: 217879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786107699 with bad export cookie 10907848212441337489 [ 7262.084096] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7262.087765] LustreError: 217879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7268.353590] Lustre: server umount lustre-MDT0001 complete [ 7295.539446] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:02:11 (1786107731) [ 7303.191744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.585281] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7303.655761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7310.595733] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7324.113532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.422490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7324.513602] Lustre: 226527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7329.654851] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7339.648826] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7346.746649] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7355.056776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.177176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7362.198218] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7371.017616] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7381.240760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.293917] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7387.976531] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7397.470141] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7399.482516] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7404.886736] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7405.390835] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7411.512942] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7411.918853] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7419.490166] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a7890810000.ost_server_uuid 50 [ 7421.464838] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a7890810000.ost_server_uuid in IDLE state after 0 sec [ 7423.009930] Lustre: Permanently deactivating lustre-OST0001 [ 7423.046659] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7428.473708] Lustre: Permanently reactivating lustre-OST0001 [ 7428.487943] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7428.492708] Lustre: Skipped 2 previous similar messages [ 7432.165497] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.166652] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.174088] LustreError: Skipped 3 previous similar messages [ 7432.179706] Lustre: Skipped 6 previous similar messages [ 7438.253712] Lustre: server umount lustre-OST0001 complete [ 7442.414988] LustreError: 228375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.431267] LustreError: 228375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7448.581437] Lustre: server umount lustre-OST0000 complete [ 7459.369225] Lustre: server umount lustre-MDT0000 complete [ 7462.995248] LustreError: 226519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786107900 with bad export cookie 10907848212441338777 [ 7462.999443] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7463.003216] LustreError: 226519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7463.600361] Lustre: server umount lustre-MDT0001 complete [ 7490.304626] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:05:26 (1786107926) [ 7498.001197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.393250] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7498.431631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7498.442360] Lustre: Skipped 2 previous similar messages [ 7498.642339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.646353] Lustre: Skipped 9 previous similar messages [ 7504.484037] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7514.996846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.204199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7515.294414] Lustre: 233226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7521.590994] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7530.721909] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7537.587774] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7545.600213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7545.844351] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7545.902964] Lustre: Permanently deactivating lustre-OST0000 [ 7545.918761] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7551.585322] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7554.981922] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7565.424473] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.732811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7573.829384] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7582.798722] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7583.560886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7593.218513] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7595.596935] Lustre: Permanently reactivating lustre-OST0000 [ 7595.613535] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7595.618842] Lustre: Skipped 2 previous similar messages [ 7601.249674] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7601.263097] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 50 seconds. I think it's dead, and I am evicting it. exp 0000000011d74caf, cur 1786108038 deadline 1786108018 last 1786107988 [ 7601.353348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7617.856188] Lustre: server umount lustre-OST0001 complete [ 7629.058361] Lustre: server umount lustre-OST0000 complete [ 7639.559769] Lustre: server umount lustre-MDT0000 complete [ 7644.555379] LustreError: 233218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108081 with bad export cookie 10907848212441340450 [ 7644.561714] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7644.573553] LustreError: 233218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7645.118813] Lustre: server umount lustre-MDT0001 complete [ 7659.370974] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7663.511221] Key type lgssc unregistered [ 7664.070563] LNet: 238114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7664.085303] LNetError: 238114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7664.114882] LNet: Removed LNI 192.168.202.150@tcp [ 7665.423195] Key type .llcrypt unregistered [ 7665.424915] Key type ._llcrypt unregistered [ 7694.597413] Key type ._llcrypt registered [ 7694.603957] Key type .llcrypt registered [ 7694.806633] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7696.524727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7696.925674] alg: No test for adler32 (adler32-zlib) [ 7698.187395] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 7698.570106] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7700.297911] Key type lgssc registered [ 7701.756381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7715.910166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7715.960306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7717.526907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.406628] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7727.084830] LustreError: 239593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.107735] LustreError: 239593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7732.203395] LustreError: 239594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.674730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.866923] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7733.922759] Lustre: 239582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7733.934249] Lustre: Permanently deactivating lustre-MDT0001 [ 7733.964635] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7734.484444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7740.222517] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7740.451447] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7749.332353] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7756.360104] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7764.545655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7765.168764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.025754] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7779.502464] LustreError: 241436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7779.530463] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7784.391188] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7784.950058] LustreError: 241436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.004126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7792.119674] LustreError: 241436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.150776] LustreError: 241436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7794.758732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.115366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7800.331648] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7800.348823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7801.629658] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7812.454242] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7817.258973] Lustre: Permanently reactivating lustre-MDT0001 [ 7817.273836] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7817.278700] Lustre: Skipped 2 previous similar messages [ 7818.716305] Lustre: lustre-MDT0001: Client 4fb6ebb6-a2ec-42eb-8386-151879ae573e (at 192.168.202.50@tcp) reconnecting [ 7826.560362] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.579112] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7826.592130] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7826.611309] LustreError: 243339:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7826.612017] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7826.624245] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7826.627057] LustreError: 243339:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7826.627212] LustreError: 243339:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.150@tcp: cfg command failed: rc = -108 [ 7826.627234] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7826.627234] [ 7826.627911] LustreError: 239586:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7828.757480] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7836.865350] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7837.148026] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7843.165899] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7843.413151] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7846.805883] Lustre: Permanently deactivating lustre-MDT0001 [ 7846.838835] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7846.846613] Lustre: Skipped 2 previous similar messages [ 7854.260468] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7867.380374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7867.405822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.423552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.202208] Lustre: server umount lustre-MDT0000 complete [ 7872.995732] LustreError: 242725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.018411] LustreError: 242725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7876.503716] LustreError: 243145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108313 with bad export cookie 11988938685359516614 [ 7876.509216] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7876.527664] LustreError: 243145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7877.265632] Lustre: server umount lustre-MDT0001 complete [ 7883.179570] Lustre: server umount lustre-OST0000 complete [ 7896.920587] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:12:12 (1786108332) [ 7911.618592] Lustre: server umount lustre-OST0001 complete [ 7921.513624] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 7934.191909] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7983.154924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7983.611606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7983.624486] Lustre: Skipped 2 previous similar messages [ 7983.704783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7983.973184] Lustre: lustre-MDT0000: new disk, initializing [ 7984.128533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.184434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7991.148873] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8003.936627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.075619] Lustre: 247864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8004.120350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8004.124902] Lustre: Skipped 1 previous similar message [ 8004.219630] Lustre: lustre-MDT0001: new disk, initializing [ 8004.284043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.336289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8004.349152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8009.854741] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8023.121889] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8030.668941] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8040.107920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.447564] Lustre: lustre-OST0000: new disk, initializing [ 8040.452582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8040.459197] Lustre: 249833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8040.561799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.266372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8047.286640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8047.550035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8048.883853] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8062.053771] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.753369] LustreError: 247870:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8075.832505] LustreError: 247870:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8075.865753] LustreError: 247872:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8076.286639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8076.458171] Lustre: lustre-OST0001: new disk, initializing [ 8076.461955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8076.468275] Lustre: 250948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8076.623242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8085.288318] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8085.935148] LustreError: 247872:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8086.128599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8086.143826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8086.332425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8096.895450] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8099.153774] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8106.981878] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.005889] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8107.013168] Lustre: Skipped 4 previous similar messages [ 8107.023435] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8107.066570] Lustre: Skipped 4 previous similar messages [ 8108.646575] Lustre: server umount lustre-OST0001 complete [ 8112.097364] LustreError: 249818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.127322] LustreError: 249818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8129.521499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8129.525714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8129.541034] LustreError: Skipped 1 previous similar message [ 8129.550715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8129.577068] Lustre: Skipped 1 previous similar message [ 8129.607087] Lustre: Skipped 1 previous similar message [ 8132.585054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8132.599265] Lustre: Skipped 1 previous similar message [ 8135.535730] Lustre: server umount lustre-OST0000 complete [ 8142.826483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8142.828843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8142.832305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8142.868458] Lustre: Skipped 1 previous similar message [ 8147.017311] Lustre: server umount lustre-MDT0000 complete [ 8147.936529] LustreError: 247875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8147.971931] LustreError: 247875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8155.128437] LustreError: 248669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108592 with bad export cookie 11988938685359519428 [ 8155.144473] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8155.154546] LustreError: 248669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8161.845617] Lustre: server umount lustre-MDT0001 complete [ 8178.698490] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8181.711794] Key type lgssc unregistered [ 8182.196784] LNet: 252986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8182.212642] LNetError: 252986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8182.275405] LNet: Removed LNI 192.168.202.150@tcp [ 8183.394929] Key type .llcrypt unregistered [ 8183.396744] Key type ._llcrypt unregistered [ 8232.033219] Key type ._llcrypt registered [ 8232.038964] Key type .llcrypt registered [ 8232.116121] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8233.279146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8233.618213] alg: No test for adler32 (adler32-zlib) [ 8234.976747] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 8235.295022] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 8237.087314] Key type lgssc registered [ 8239.275808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8248.889329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8248.903512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.140498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8250.166919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8250.485133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.804681] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8268.084829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.299385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8268.349871] Lustre: 255199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8268.749086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.545763] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8282.563666] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8292.725220] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8303.580986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8303.962468] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8304.159692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8309.873821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8311.849811] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8322.420653] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8382.966782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.981524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8382.996038] Lustre: Skipped 1 previous similar message [ 8383.019996] Lustre: Skipped 1 previous similar message [ 8384.795956] Lustre: server umount lustre-OST0000 complete [ 8390.677937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8406.615780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8407.108690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8416.320181] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8423.475551] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8423.500756] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8423.506543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8427.401192] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8469.474162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.497783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8469.504359] Lustre: Skipped 1 previous similar message [ 8469.524684] Lustre: Skipped 1 previous similar message [ 8474.342755] Lustre: server umount lustre-OST0000 complete [ 8479.291895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.300592] Lustre: Skipped 1 previous similar message [ 8479.302218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8484.320902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8484.329119] Lustre: Skipped 2 previous similar messages [ 8485.885277] Lustre: server umount lustre-MDT0000 complete [ 8491.628503] LustreError: 255191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786108928 with bad export cookie 4586751221262781756 [ 8491.637812] LustreError: 255191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8491.641840] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8492.431235] Lustre: server umount lustre-MDT0001 complete [ 8508.204096] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8512.749938] Key type lgssc unregistered [ 8513.508756] LNet: 260634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8513.524705] LNetError: 260634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8513.599904] LNet: Removed LNI 192.168.202.150@tcp [ 8515.798187] Key type .llcrypt unregistered [ 8515.801783] Key type ._llcrypt unregistered [ 8554.318984] Key type ._llcrypt registered [ 8554.323260] Key type .llcrypt registered [ 8554.455340] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8555.767475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8555.835598] alg: No test for adler32 (adler32-zlib) [ 8557.052978] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 8557.426237] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 8559.143370] Key type lgssc registered [ 8560.595055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8570.654220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8570.671647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.327626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8577.751561] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8582.631218] LustreError: 262057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.664419] LustreError: 262057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8587.748671] LustreError: 262058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.106770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.964560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8595.458693] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8603.971965] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8611.609902] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8619.478900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8619.748799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.814574] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8634.679383] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8643.399611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8669.160120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.163428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8669.172322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8669.184872] Lustre: Skipped 1 previous similar message [ 8672.336633] Lustre: server umount lustre-OST0000 complete [ 8676.835791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8676.843357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.850387] Lustre: Skipped 1 previous similar message [ 8676.853299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.393563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.394285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.208392] Lustre: server umount lustre-MDT0000 complete [ 8684.519432] LustreError: 262057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.540666] LustreError: 262057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8685.241460] LustreError: 262791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109122 with bad export cookie 6911856513352259589 [ 8685.251204] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8685.254290] LustreError: 262791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8685.659397] Lustre: server umount lustre-MDT0001 complete [ 8695.158242] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8697.465638] Key type lgssc unregistered [ 8697.706713] LNet: 266386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8697.712303] LNetError: 266386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8698.794849] LNet: Removed LNI 192.168.202.150@tcp [ 8699.484574] Key type .llcrypt unregistered [ 8699.486076] Key type ._llcrypt unregistered [ 8712.898199] Key type ._llcrypt registered [ 8712.902329] Key type .llcrypt registered [ 8712.982852] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8713.778472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8713.865066] alg: No test for adler32 (adler32-zlib) [ 8714.933297] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 8715.110818] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 8716.719346] Key type lgssc registered [ 8717.434513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8722.591730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8722.604677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.882994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8726.804364] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8733.153964] LustreError: 267587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8733.165060] LustreError: 267587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8733.370075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.632552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8737.149672] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8742.794380] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8747.025423] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8752.916883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8753.287400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.138324] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8763.720779] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8768.622140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8779.234956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.241602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.244267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8779.250872] Lustre: Skipped 1 previous similar message [ 8782.030108] Lustre: server umount lustre-OST0000 complete [ 8784.352150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.363401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8784.367411] Lustre: Skipped 2 previous similar messages [ 8789.474373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8789.482295] Lustre: Skipped 1 previous similar message [ 8790.610564] Lustre: server umount lustre-MDT0000 complete [ 8793.083965] LustreError: 267569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109230 with bad export cookie 12818086945045706821 [ 8793.087453] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8793.099727] LustreError: 267569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8793.287747] Lustre: server umount lustre-MDT0001 complete [ 8799.973549] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8801.632560] Key type lgssc unregistered [ 8801.850219] LNet: 271263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8801.865464] LNetError: 271263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8802.917704] LNet: Removed LNI 192.168.202.150@tcp [ 8803.444771] Key type .llcrypt unregistered [ 8803.446336] Key type ._llcrypt unregistered [ 8818.964294] Key type ._llcrypt registered [ 8818.966296] Key type .llcrypt registered [ 8819.028446] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8819.775228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8819.794162] alg: No test for adler32 (adler32-zlib) [ 8820.683063] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 8820.856384] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 8822.745766] Key type lgssc registered [ 8824.391360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8835.110373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8835.119503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8836.857267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.371287] LustreError: 272660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.412291] LustreError: 272660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8846.643792] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8851.432539] LustreError: 272659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8856.551667] LustreError: 272660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.667499] LustreError: 272659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8862.548447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.371373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8869.615659] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8883.078943] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8890.266537] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8899.510694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.162341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8906.322426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8907.785538] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8916.523681] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8944.102539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.104494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.108359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.114490] Lustre: Skipped 1 previous similar message [ 8944.135530] Lustre: Skipped 1 previous similar message [ 8949.218672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8949.223263] Lustre: Skipped 1 previous similar message [ 8950.499764] Lustre: server umount lustre-OST0000 complete [ 8955.362457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8955.374368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.399671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8959.456982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8959.457941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8959.476014] Lustre: Skipped 1 previous similar message [ 8961.445209] Lustre: server umount lustre-MDT0000 complete [ 8964.582868] LustreError: 272653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8964.601026] LustreError: 272653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8965.169245] LustreError: 272639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109402 with bad export cookie 13158193558028824659 [ 8965.174924] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.527332] Lustre: server umount lustre-MDT0001 complete [ 8976.898709] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8979.472643] Key type lgssc unregistered [ 8979.787910] LNet: 276977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8979.793823] LNetError: 276977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8979.814914] LNet: Removed LNI 192.168.202.150@tcp [ 8980.574821] Key type .llcrypt unregistered [ 8980.576606] Key type ._llcrypt unregistered [ 8997.618282] Key type ._llcrypt registered [ 8997.620089] Key type .llcrypt registered [ 8997.735651] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8999.020523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8999.255602] alg: No test for adler32 (adler32-zlib) [ 9000.579945] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 9000.976733] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 9002.783231] Key type lgssc registered [ 9004.301135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9013.421937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9013.455965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.135081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.727063] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9020.399223] LustreError: 278210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.416568] LustreError: 278210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9025.507557] LustreError: 278212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9028.855939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.275380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9034.696345] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9042.719118] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9048.891829] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9056.582515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9057.028675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.115521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9063.945600] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9071.191871] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9085.920549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9085.927652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.950131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9090.024471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9090.034876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9090.045460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9091.930228] Lustre: server umount lustre-OST0000 complete [ 9100.257783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9100.274592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9100.280235] Lustre: Skipped 2 previous similar messages [ 9102.326337] Lustre: server umount lustre-MDT0000 complete [ 9105.385373] LustreError: 278219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9105.406316] LustreError: 278219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9106.356000] LustreError: 278970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109543 with bad export cookie 13481380216052796358 [ 9106.386754] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9106.396884] LustreError: 278970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9106.994916] Lustre: server umount lustre-MDT0001 complete [ 9120.284316] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 9122.732667] Key type lgssc unregistered [ 9123.024885] LNet: 281914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9123.029896] LNetError: 281914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9124.074883] LNet: Removed LNI 192.168.202.150@tcp [ 9124.893207] Key type .llcrypt unregistered [ 9124.898564] Key type ._llcrypt unregistered [ 9137.877751] Key type ._llcrypt registered [ 9137.881695] Key type .llcrypt registered [ 9138.017741] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9156.986850] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 9172.754557] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9174.148228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9174.508160] alg: No test for adler32 (adler32-zlib) [ 9175.785706] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 9176.111570] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 9177.839209] Key type lgssc registered [ 9179.152351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9219.141856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9219.188571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9220.582714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9220.637038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9220.747842] Lustre: lustre-MDT0000: new disk, initializing [ 9220.870275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9220.901844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9225.700973] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9237.549450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.718117] Lustre: 285470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9237.785584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9237.788741] Lustre: Skipped 1 previous similar message [ 9237.867102] Lustre: lustre-MDT0001: new disk, initializing [ 9237.982376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9238.039808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9238.055557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9243.486596] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9254.918244] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9261.386189] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9268.598239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9268.885548] Lustre: lustre-OST0000: new disk, initializing [ 9268.888672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9268.896770] Lustre: 287440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9269.000567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.267524] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9279.041359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9279.052552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9279.227130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9287.691197] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9293.121698] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9293.347463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9298.919853] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9299.164222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9305.059032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9305.066706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9305.071985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9305.073032] LustreError: Skipped 1 previous similar message [ 9305.101976] Lustre: Skipped 1 previous similar message [ 9307.477251] Lustre: server umount lustre-OST0000 complete [ 9314.785492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9314.793176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.819256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.828085] Lustre: Skipped 1 previous similar message [ 9318.189692] Lustre: server umount lustre-MDT0000 complete [ 9322.569192] LustreError: 285463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109759 with bad export cookie 14715850396782121987 [ 9322.572949] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9322.603675] LustreError: 285463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9323.154757] Lustre: server umount lustre-MDT0001 complete [ 9333.361528] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:36:09 (1786109769) [ 9340.858529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.417746] LustreError: 289854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.438864] LustreError: 289854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9341.488877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9346.544203] LustreError: 289855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9346.641839] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9351.654483] LustreError: 289854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.053226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9356.470089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9361.290172] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9370.110167] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9377.381583] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9386.184769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9386.538625] LustreError: 291676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.790199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9395.311309] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9403.431759] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9412.577809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9412.581465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.591157] Lustre: Skipped 1 previous similar message [ 9412.594672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.597983] Lustre: Skipped 2 previous similar messages [ 9416.673082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9416.678344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9416.688803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.365600] Lustre: server umount lustre-OST0000 complete [ 9423.330676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9423.345986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.375089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.382020] Lustre: Skipped 1 previous similar message [ 9429.525662] Lustre: server umount lustre-MDT0000 complete [ 9432.047175] LustreError: 290859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.072439] LustreError: 290859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9433.392377] LustreError: 291698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786109870 with bad export cookie 14715850396782122869 [ 9433.397552] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9433.400368] LustreError: 291698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9433.835704] Lustre: server umount lustre-MDT0001 complete [ 9445.940493] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 9448.768500] Key type lgssc unregistered [ 9449.099556] LNet: 293393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9449.109886] LNetError: 293393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9449.132932] LNet: Removed LNI 192.168.202.150@tcp [ 9449.958924] Key type .llcrypt unregistered [ 9449.961237] Key type ._llcrypt unregistered [ 9477.570733] Key type ._llcrypt registered [ 9477.572090] Key type .llcrypt registered [ 9477.687947] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 9491.404270] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9492.525279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9492.543795] alg: No test for adler32 (adler32-zlib) [ 9493.525580] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 9493.739108] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 9495.468059] Key type lgssc registered [ 9496.695547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9544.761517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9544.832140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9546.082538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9546.103223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9546.174293] Lustre: lustre-MDT0000: new disk, initializing [ 9546.255082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9546.316482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9552.094173] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9562.470192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.616571] Lustre: 297215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9562.693622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9562.696444] Lustre: Skipped 1 previous similar message [ 9562.804530] Lustre: lustre-MDT0001: new disk, initializing [ 9562.913390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.954123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9562.967251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9567.577199] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9577.240852] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9583.374432] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.741296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.941476] Lustre: lustre-OST2710: new disk, initializing [ 9590.944634] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9590.949936] Lustre: 299184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.040882] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9598.078139] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9599.085563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9599.104274] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9599.219099] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9609.497993] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9616.995519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9617.264854] Lustre: lustre-OST03e8: new disk, initializing [ 9617.271376] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9617.278039] Lustre: 300252:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9617.393088] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9623.366499] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9627.316641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9627.331260] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9627.445350] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9631.691463] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9640.236501] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9640.467696] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9645.624335] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9645.838402] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9650.712074] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9650.924242] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9655.688995] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9655.941506] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9660.384499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9660.395701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.419881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.613928] Lustre: server umount lustre-MDT0000 complete [ 9663.472106] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.482785] Lustre: Skipped 2 previous similar messages [ 9666.866183] LustreError: 299195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110103 with bad export cookie 3981692952133046474 [ 9666.879315] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.880898] LustreError: 299195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9667.158382] Lustre: server umount lustre-MDT0001 complete [ 9681.471520] Lustre: server umount lustre-OST2710 complete [ 9683.938899] Lustre: 295011:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786110105/real 1786110105] req@00000000f25a6a75 x1872872006027136/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786110121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9683.967449] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9686.245687] Lustre: server umount lustre-OST03e8 complete [ 9705.104481] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 9716.175986] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9757.420967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.808198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9757.865488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9757.957278] Lustre: lustre-MDT0000: new disk, initializing [ 9758.088470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9758.112447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9763.317259] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9775.007814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.157356] Lustre: 305499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9775.206890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9775.211626] Lustre: Skipped 1 previous similar message [ 9775.285910] Lustre: lustre-MDT0001: new disk, initializing [ 9775.388710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9775.431071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9775.445796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9780.874913] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9792.173766] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9798.060196] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9805.632381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9806.027532] Lustre: lustre-OST0000: new disk, initializing [ 9806.030594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9806.034906] Lustre: 307471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9806.174379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9813.009680] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9814.638347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9814.650161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9814.774423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9822.789912] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9827.466387] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9827.659878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9832.462197] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9832.702338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9835.491903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9835.493580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9835.510424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9835.515267] Lustre: Skipped 1 previous similar message [ 9835.524172] Lustre: Skipped 1 previous similar message [ 9840.619453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9840.625869] Lustre: Skipped 1 previous similar message [ 9840.944765] Lustre: server umount lustre-OST0000 complete [ 9845.730093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.731145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9845.743925] Lustre: Skipped 1 previous similar message [ 9845.752120] Lustre: Skipped 1 previous similar message [ 9850.849939] LustreError: 305505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9850.861553] LustreError: 305505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9850.959867] Lustre: server umount lustre-MDT0000 complete [ 9854.479108] LustreError: 305489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110291 with bad export cookie 3981692952133048154 [ 9854.482362] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9854.492604] LustreError: 305489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9855.045804] Lustre: server umount lustre-MDT0001 complete [ 9863.133840] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:44:59 (1786110299) [ 9864.602559] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9866.148773] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:45:02 (1786110302) [ 9869.094913] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9893.732298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9894.039984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9894.066226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9894.254446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.273626] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9907.620285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9907.776102] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9907.859851] Lustre: 311326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9912.793673] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9935.214440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9935.426080] Lustre: Denying initial registration attempt from nid 192.168.202.150@tcp, specified as failover [ 9935.431234] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9935.437387] LustreError: 312780:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9935.440562] LustreError: 312780:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9935.443278] LustreError: 312780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9935.489237] Lustre: server umount lustre-OST0000 complete [ 9935.490936] LustreError: 312780:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9943.522341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.528888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9943.535420] Lustre: Skipped 1 previous similar message [ 9943.547208] Lustre: Skipped 1 previous similar message [ 9948.368541] Lustre: server umount lustre-MDT0000 complete [ 9948.640387] LustreError: 311333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.661241] LustreError: 311333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9951.833727] LustreError: 311317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110388 with bad export cookie 3981692952133049029 [ 9951.836675] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9951.846946] LustreError: 311317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9952.082712] Lustre: server umount lustre-MDT0001 complete [ 9963.154746] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 9966.236513] Key type lgssc unregistered [ 9966.497705] LNet: 313904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9966.504460] LNetError: 313904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9966.525194] LNet: Removed LNI 192.168.202.150@tcp [ 9967.366874] Key type .llcrypt unregistered [ 9967.369288] Key type ._llcrypt unregistered [ 9977.582148] Key type ._llcrypt registered [ 9977.584707] Key type .llcrypt registered [ 9977.701847] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9978.602696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9978.729924] alg: No test for adler32 (adler32-zlib) [ 9979.793139] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [ 9980.036740] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 9981.711321] Key type lgssc registered [ 9982.708839] Lustre: Echo OBD driver; http://www.lustre.org/ [10025.207308] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10033.316385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10033.352332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.719228] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10034.739165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10034.855482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10039.095205] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10046.426980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.519638] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10046.560168] Lustre: 316650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10046.846934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10050.950539] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10058.042162] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10062.278558] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10081.008496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.246701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10081.354803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10087.071190] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10094.661332] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10098.660506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10098.661795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10098.694888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10098.715975] Lustre: Skipped 1 previous similar message [10102.671180] Lustre: server umount lustre-OST0000 complete [10108.384701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10108.390490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10108.397264] Lustre: Skipped 1 previous similar message [10108.408417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10112.497309] Lustre: server umount lustre-MDT0000 complete [10115.826135] LustreError: 316641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110552 with bad export cookie 13142723820225234930 [10115.837512] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10115.839121] LustreError: 316641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10116.401614] Lustre: server umount lustre-MDT0001 complete [10126.986841] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [10129.359868] Key type lgssc unregistered [10129.638339] LNet: 320227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10129.655388] LNetError: 320227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10130.725557] LNet: Removed LNI 192.168.202.150@tcp [10131.364829] Key type .llcrypt unregistered [10131.367161] Key type ._llcrypt unregistered [10151.132955] Key type ._llcrypt registered [10151.135461] Key type .llcrypt registered [10151.249480] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10152.295282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10152.320868] alg: No test for adler32 (adler32-zlib) [10153.443294] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [10153.690457] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [10155.391213] Key type lgssc registered [10156.353655] Lustre: Echo OBD driver; http://www.lustre.org/ [10162.065390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10162.081818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.382563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10166.924505] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10172.899392] LustreError: 321626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.907290] LustreError: 321626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10174.961056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.262439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10179.004743] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10185.698804] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10190.392209] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10198.894195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10199.238988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10204.912174] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10211.344808] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10221.536118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10221.541075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.548716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.100797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.101828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10224.119234] Lustre: Skipped 1 previous similar message [10226.519035] Lustre: server umount lustre-MDT0000 complete [10229.217058] LustreError: 321625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.235099] LustreError: 321625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10229.883634] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110666 with bad export cookie 13049868081027547379 [10229.891346] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10229.896370] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10234.337324] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10234.343179] LustreError: 321620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10234.354305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10234.354313] Lustre: Skipped 3 previous similar messages [10234.387435] LustreError: 321620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10236.310816] Lustre: server umount lustre-MDT0001 complete [10242.524698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10249.739235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10255.778786] LustreError: 320934:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000094af7fa6 x1872872698060416/t0(0) o250->MGC192.168.202.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10255.996059] LustreError: 323456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.010414] LustreError: 323456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10256.052868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10256.122790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10259.703599] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10264.228209] LustreError: 323457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.233614] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10264.242760] LustreError: 323457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10266.629660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.982277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10270.615469] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10274.306279] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10277.673711] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10278.987770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10283.628751] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.061324] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10289.633969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10289.637337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10289.648174] Lustre: Skipped 1 previous similar message [10289.659910] Lustre: Skipped 2 previous similar messages [10294.756176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10294.763843] Lustre: Skipped 1 previous similar message [10295.249870] Lustre: server umount lustre-OST0000 complete [10299.874090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10299.884519] Lustre: Skipped 1 previous similar message [10304.993844] LustreError: 325224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10305.006730] LustreError: 325224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10305.059624] Lustre: server umount lustre-MDT0000 complete [10308.401146] LustreError: 321606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110745 with bad export cookie 13049868081027555030 [10308.405476] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10308.407653] LustreError: 321606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10308.853595] Lustre: server umount lustre-MDT0001 complete [10317.800141] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [10319.989203] Key type lgssc unregistered [10320.253445] LNet: 327893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10320.262369] LNetError: 327893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10320.277528] LNet: Removed LNI 192.168.202.150@tcp [10320.924469] Key type .llcrypt unregistered [10320.926183] Key type ._llcrypt unregistered [10338.294883] Key type ._llcrypt registered [10338.301961] Key type .llcrypt registered [10338.453152] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10339.553816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10339.602383] alg: No test for adler32 (adler32-zlib) [10340.597263] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [10340.824647] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [10342.463162] Key type lgssc registered [10343.422731] Lustre: Echo OBD driver; http://www.lustre.org/ [10350.128212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10350.152237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10351.601228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10355.382940] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10361.825276] LustreError: 329288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10361.835612] LustreError: 329288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10362.311282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.592676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10366.093464] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10372.799798] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10377.032874] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10381.280835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.291925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10385.542331] Lustre: server umount lustre-MDT0000 complete [10386.405179] LustreError: 329283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.425126] LustreError: 329283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10387.424994] LustreError: 329288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10387.452849] LustreError: 329288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10388.785813] LustreError: 329267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110825 with bad export cookie 12600651495298635204 [10388.803981] LustreError: 329267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10388.804372] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.184509] Lustre: server umount lustre-MDT0001 complete [10402.167048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.475320] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10402.497873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10402.643488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.476610] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10413.673272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.789842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10413.815836] Lustre: 331874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10414.075545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10417.335316] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10422.498291] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10426.744539] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10434.670243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.817789] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10434.825763] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10434.830800] LustreError: 333758:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10434.834713] LustreError: 333758:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10434.839212] LustreError: 333758:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10434.890524] Lustre: server umount lustre-OST0000 complete [10434.892616] LustreError: 333758:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10443.631764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10443.771198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10443.918376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.994255] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10452.065874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10453.597572] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10458.944146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10459.004967] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10459.070615] Lustre: lustre-OST0001: new disk, initializing [10459.074186] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10459.077971] Lustre: 335299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10459.133992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10459.834130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10459.839749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10459.948155] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10463.240710] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10468.612414] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10475.492136] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10475.504315] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10475.509611] Lustre: Skipped 2 previous similar messages [10475.509642] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10475.509647] Lustre: Skipped 1 previous similar message [10475.514173] LustreError: Skipped 1 previous similar message [10477.130707] Lustre: server umount lustre-OST0001 complete [10480.608802] LustreError: 334225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10480.632925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10480.636733] LustreError: 334225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10480.644448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10480.660668] Lustre: Skipped 1 previous similar message [10480.686200] Lustre: Skipped 2 previous similar messages [10485.734795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10485.737193] LustreError: 334224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.738688] Lustre: Skipped 1 previous similar message [10485.757934] LustreError: 334224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10486.616616] Lustre: server umount lustre-OST0000 complete [10490.851092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.853314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10490.866438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.873866] Lustre: Skipped 1 previous similar message [10496.255262] Lustre: server umount lustre-MDT0000 complete [10498.733032] LustreError: 331866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786110935 with bad export cookie 12600651495298635757 [10498.741628] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10498.745487] LustreError: 331866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10498.994583] Lustre: server umount lustre-MDT0001 complete [10506.690872] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [10508.567934] Key type lgssc unregistered [10508.776512] LNet: 337260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10508.779690] LNetError: 337260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10508.789874] LNet: Removed LNI 192.168.202.150@tcp [10509.334095] Key type .llcrypt unregistered [10509.336478] Key type ._llcrypt unregistered [10563.073506] Key type ._llcrypt registered [10563.075961] Key type .llcrypt registered [10563.175606] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [10572.352033] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10572.934700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10572.960985] alg: No test for adler32 (adler32-zlib) [10573.848836] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [10573.959298] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [10575.551258] Key type lgssc registered [10576.052321] Lustre: Echo OBD driver; http://www.lustre.org/ [10600.906869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10600.921586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.186446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10602.227591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10602.324533] Lustre: lustre-MDT0000: new disk, initializing [10602.388523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10602.398692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10606.297651] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10615.210574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10615.280274] Lustre: 342166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10615.323042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10615.326859] Lustre: Skipped 1 previous similar message [10615.376487] Lustre: lustre-MDT0001: new disk, initializing [10615.430777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10615.474610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10615.493654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10618.747527] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10625.290711] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10629.246334] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10634.602642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10634.792963] Lustre: lustre-OST0000: new disk, initializing [10634.796387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10634.801242] Lustre: 344134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10634.875967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10639.765648] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10640.931716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10640.937311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10640.990622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10647.334455] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10650.977120] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10651.172826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10655.011530] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10655.235579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10661.346419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10661.347963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10661.358382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10661.379216] Lustre: Skipped 1 previous similar message [10662.887665] Lustre: server umount lustre-OST0000 complete [10666.464186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10666.478065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10666.481828] Lustre: Skipped 2 previous similar messages [10671.589971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10671.598302] Lustre: Skipped 1 previous similar message [10672.343774] Lustre: server umount lustre-MDT0000 complete [10675.321083] LustreError: 342158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111112 with bad export cookie 11297739952742276942 [10675.330036] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10675.333521] LustreError: 342158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10675.638528] Lustre: server umount lustre-MDT0001 complete [10682.085591] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:58:38 (1786111118) [10687.881858] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:58:44 (1786111124) [10694.967718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.367524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10698.573984] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10700.774598] LustreError: 346887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.786041] LustreError: 346887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10705.245536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10705.541375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10709.050780] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10715.209661] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.429504] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10724.175878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10724.390240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10728.744281] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10733.785615] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10750.453251] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10751.697687] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10752.906944] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10754.016656] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10755.390710] Lustre: Failing over lustre-MDT0000 [10755.621143] Lustre: server umount lustre-MDT0000 complete [10755.734144] LustreError: 346883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.576164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10756.586142] LustreError: Skipped 1 previous similar message [10756.589067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.599640] Lustre: Skipped 1 previous similar message [10757.091057] LustreError: 347614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10757.103023] LustreError: 347614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10760.828156] LustreError: 346882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.951879] LustreError: 346882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.967783] LustreError: 346882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10771.621408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.713882] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10771.871860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10771.897523] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10772.124337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10774.724455] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10777.061183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10777.073343] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10777.098758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10781.194858] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10782.287894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10784.794863] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10786.039705] Lustre: Failing over lustre-MDT0000 [10786.239382] Lustre: server umount lustre-MDT0000 complete [10787.300508] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10787.308226] LustreError: 346887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.313324] Lustre: Skipped 3 previous similar messages [10787.317816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10787.334561] LustreError: 346887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10801.458088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.545855] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10801.728236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10802.809050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10803.941816] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10806.765503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10806.770274] Lustre: Skipped 2 previous similar messages [10806.785357] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10806.810809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10810.064088] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10810.906410] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10816.993346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10817.003767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.007205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10817.037869] Lustre: Skipped 3 previous similar messages [10820.438477] Lustre: server umount lustre-MDT0000 complete [10822.119061] LustreError: 346887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10822.140139] LustreError: 346887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10823.007759] LustreError: 346867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111260 with bad export cookie 11297739952742279651 [10823.013390] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10823.016621] LustreError: 346867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10823.291093] Lustre: server umount lustre-MDT0001 complete [10836.442867] Lustre: server umount lustre-OST0000 complete [10847.294547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.737841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.742832] Lustre: Skipped 1 previous similar message [10850.873230] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10856.936069] LustreError: 353253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.958313] LustreError: 353253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10857.116454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.017892] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10866.473674] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10870.333287] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10875.184681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10879.702359] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10885.091704] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10892.792521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10898.990982] Lustre: server umount lustre-OST0000 complete [10903.009172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.010984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10903.018606] Lustre: Skipped 1 previous similar message [10903.022248] Lustre: Skipped 2 previous similar messages [10911.243383] Lustre: server umount lustre-MDT0000 complete [10913.845922] LustreError: 353987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111350 with bad export cookie 11297739952742280400 [10913.848324] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.859723] LustreError: 353987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10923.548965] Lustre: server umount lustre-MDT0001 complete [10929.538825] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [10931.326864] Key type lgssc unregistered [10931.583667] LNet: 356805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10931.591173] LNetError: 356805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10931.610187] LNet: Removed LNI 192.168.202.150@tcp [10932.161293] Key type .llcrypt unregistered [10932.167373] Key type ._llcrypt unregistered [10948.755181] Key type ._llcrypt registered [10948.756706] Key type .llcrypt registered [10948.822825] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10949.256578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10949.297341] alg: No test for adler32 (adler32-zlib) [10950.216866] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [10950.374083] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [10952.007706] Key type lgssc registered [10952.694827] Lustre: Echo OBD driver; http://www.lustre.org/ [10957.233043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10957.243849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.496080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10961.542455] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10966.778416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.998505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10969.715228] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10973.896897] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10977.242980] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10981.193696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10981.395866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10983.464107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10984.602425] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10988.573943] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11002.850958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.852298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11002.852774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11002.856451] Lustre: Skipped 1 previous similar message [11004.804870] Lustre: server umount lustre-MDT0000 complete [11006.586875] LustreError: 359034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111443 with bad export cookie 4295058809077934942 [11006.593075] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11006.744778] Lustre: server umount lustre-MDT0001 complete [11018.874689] Lustre: server umount lustre-OST0000 complete [11026.959479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.190545] LustreError: 362006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11027.197278] LustreError: 362006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11027.231591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11029.475916] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11032.544858] LustreError: 362007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11034.443155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.669766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11037.027654] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11041.110388] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11043.999991] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11048.793451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11048.991954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11052.088354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11056.364753] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11069.371341] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11085.826105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.836954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.849436] Lustre: Skipped 2 previous similar messages [11085.878782] Lustre: Skipped 4 previous similar messages [11087.514621] Lustre: server umount lustre-MDT0000 complete [11090.922387] LustreError: 362006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.959344] LustreError: 362006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11092.952045] LustreError: 361986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111530 with bad export cookie 4295058809077936055 [11092.957131] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11092.959577] LustreError: 361986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11093.371059] Lustre: server umount lustre-MDT0001 complete [11111.150473] Lustre: server umount lustre-OST0000 complete [11133.970986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.507280] LustreError: 365923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.566383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11139.854505] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11140.088350] LustreError: 365924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.289595] LustreError: 365923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.313882] LustreError: 365923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11150.933465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11157.362311] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11166.324653] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11173.146577] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11180.742954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11181.047233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11181.050364] Lustre: Skipped 1 previous similar message [11188.137680] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11194.947860] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11204.603499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11211.745569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11211.758811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11211.777497] Lustre: Skipped 1 previous similar message [11211.785807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11214.817303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11214.818413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11214.833710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11216.474663] Lustre: server umount lustre-OST0000 complete [11223.009786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11223.019905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11223.044446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11223.050531] Lustre: Skipped 1 previous similar message [11227.122412] Lustre: server umount lustre-MDT0000 complete [11230.177929] LustreError: 368416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.198878] LustreError: 368416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11231.927617] LustreError: 365904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111669 with bad export cookie 4295058809077937077 [11231.944430] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11231.945405] LustreError: 365904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11232.483385] Lustre: server umount lustre-MDT0001 complete [11245.570709] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11248.714661] Key type lgssc unregistered [11248.988642] LNet: 369483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11248.994136] LNetError: 369483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11249.009772] LNet: Removed LNI 192.168.202.150@tcp [11250.053779] Key type .llcrypt unregistered [11250.055956] Key type ._llcrypt unregistered [11276.076436] Key type ._llcrypt registered [11276.078449] Key type .llcrypt registered [11276.184675] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11277.179977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11277.321662] alg: No test for adler32 (adler32-zlib) [11278.468781] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [11278.759329] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11280.407291] Key type lgssc registered [11281.799991] Lustre: Echo OBD driver; http://www.lustre.org/ [11289.990139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11290.013015] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.031234] LustreError: 370924:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11290.049219] LustreError: 370924:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11290.056498] LustreError: 370924:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11290.063321] LustreError: 370924:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11290.077823] LustreError: 370924:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11298.182737] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11298.189804] LustreError: 371321:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11298.202431] LustreError: 371321:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11298.213251] LustreError: 371321:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11298.218813] LustreError: 371321:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11298.227887] LustreError: 371321:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11318.347959] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11320.971869] Key type lgssc unregistered [11321.335525] LNet: 372326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11321.344478] LNetError: 372326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11321.368299] LNet: Removed LNI 192.168.202.150@tcp [11322.128960] Key type .llcrypt unregistered [11322.132287] Key type ._llcrypt unregistered [11342.441257] Key type ._llcrypt registered [11342.444643] Key type .llcrypt registered [11342.579842] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [11357.656797] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11359.280635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11359.610032] alg: No test for adler32 (adler32-zlib) [11360.766080] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [11361.087360] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11362.792313] Key type lgssc registered [11364.078933] Lustre: Echo OBD driver; http://www.lustre.org/ [11402.833477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11402.875245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11404.274994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11404.358625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11404.444409] Lustre: lustre-MDT0000: new disk, initializing [11404.515167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11404.538865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11410.259324] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11421.785671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11422.062905] Lustre: 375487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11422.146516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11422.161950] Lustre: Skipped 1 previous similar message [11422.294550] Lustre: lustre-MDT0001: new disk, initializing [11422.413480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11422.477633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11422.492071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11426.933185] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11437.415502] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11443.024886] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11449.942959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11450.309633] Lustre: lustre-OST0000: new disk, initializing [11450.318297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11450.327474] Lustre: 377456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11450.433437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11457.067794] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11458.117522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11458.130404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11458.246502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11467.137141] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11471.996080] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11472.275773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11477.516138] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11477.956774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11484.138727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11484.141331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11484.151341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11484.186046] Lustre: Skipped 1 previous similar message [11489.254292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11489.261400] Lustre: Skipped 1 previous similar message [11494.367194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11494.372911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11494.380059] Lustre: Skipped 1 previous similar message [11494.537237] Lustre: server umount lustre-OST0000 complete [11499.491725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11499.492328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11499.505495] LustreError: Skipped 1 previous similar message [11499.511524] Lustre: Skipped 1 previous similar message [11499.519411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11499.534962] Lustre: Skipped 1 previous similar message [11504.984637] Lustre: server umount lustre-MDT0000 complete [11508.287812] LustreError: 375478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786111945 with bad export cookie 1278514553370846817 [11508.289049] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11508.302910] LustreError: 375478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11508.688189] Lustre: server umount lustre-MDT0001 complete [11517.454893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11519.723543] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:12:35 (1786111955) [11525.035978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11525.402038] LustreError: 379917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.419660] LustreError: 379917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11525.478986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11529.751556] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11530.738267] LustreError: 379918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11535.850150] LustreError: 379918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.984327] LustreError: 379917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.781087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.316245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11548.279542] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11555.840632] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11563.016717] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11571.661911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11571.971395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11578.545404] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11586.141519] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11593.358787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11593.498935] Lustre: lustre-OST0001: new disk, initializing [11593.501634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11593.508396] Lustre: 382752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11593.592528] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11599.811880] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11602.585451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11602.610158] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11602.730471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11609.538440] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11616.976551] Lustre: lustre-OST0001: Not available for connect from 192.168.202.50@tcp (stopping) [11618.278532] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11618.285419] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11618.287428] LustreError: Skipped 1 previous similar message [11618.320331] Lustre: Skipped 2 previous similar messages [11620.519949] Lustre: server umount lustre-OST0001 complete [11622.040309] LustreError: 381738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.643338] LustreError: 381740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.671035] LustreError: 381740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11641.827420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11641.835524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.846409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.853179] Lustre: Skipped 2 previous similar messages [11645.354873] Lustre: server umount lustre-OST0000 complete [11650.043344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11650.064308] LustreError: Skipped 1 previous similar message [11650.066033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.094589] Lustre: Skipped 1 previous similar message [11656.646146] Lustre: server umount lustre-MDT0000 complete [11658.213596] LustreError: 379918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11658.230678] LustreError: 379918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11660.427567] LustreError: 379898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112097 with bad export cookie 1278514553370847699 [11660.429808] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11660.435466] LustreError: 379898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11666.857236] Lustre: server umount lustre-MDT0001 complete [11680.808195] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11683.733194] Key type lgssc unregistered [11684.275676] LNet: 384741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.286226] LNetError: 384741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.304581] LNet: Removed LNI 192.168.202.150@tcp [11685.636032] Key type .llcrypt unregistered [11685.640700] Key type ._llcrypt unregistered [11741.453449] Key type ._llcrypt registered [11741.455260] Key type .llcrypt registered [11741.560468] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11742.739498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11742.974158] alg: No test for adler32 (adler32-zlib) [11744.158385] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [11744.330684] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11745.967153] Key type lgssc registered [11747.280133] Lustre: Echo OBD driver; http://www.lustre.org/ [11755.909336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11755.924545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.084311] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11757.097310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11757.167455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11762.740402] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11773.339134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.456455] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11773.492244] Lustre: 387543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11773.819430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11779.946143] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11787.626449] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11793.722661] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11799.520260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.525528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11799.546081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11799.560958] Lustre: Skipped 1 previous similar message [11801.886343] Lustre: server umount lustre-MDT0000 complete [11805.585833] LustreError: 387534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112242 with bad export cookie 4837715883519806388 [11805.593774] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.596359] LustreError: 387534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11805.902397] Lustre: server umount lustre-MDT0001 complete [11814.725915] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:17:31 (1786112251) [11821.572788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.946606] LustreError: 390057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.970511] LustreError: 390057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11822.041703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11826.562201] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11827.175225] LustreError: 390058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11827.187877] LustreError: 390058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11832.291829] LustreError: 390057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.067155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.279710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11840.043893] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11847.827837] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11853.862146] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11861.098666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11861.360630] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11861.577465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11867.959540] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11876.517319] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11885.101136] Lustre: Permanently deactivating lustre-OST0000 [11885.121808] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11892.837445] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11895.174672] LustreError: 393045:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11895.179692] LustreError: 393045:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11895.182553] LustreError: 393045:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11897.647876] LustreError: 393094:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11897.653314] LustreError: 393094:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11906.050128] Lustre: server umount lustre-OST0000 complete [11910.605257] LustreError: 393343:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11910.622283] LustreError: 393343:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11913.185474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11913.194794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11913.207258] Lustre: Skipped 2 previous similar messages [11913.224969] Lustre: Skipped 1 previous similar message [11918.306041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11918.314098] Lustre: Skipped 1 previous similar message [11923.429143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11923.446167] Lustre: Skipped 1 previous similar message [11923.624521] Lustre: server umount lustre-MDT0000 complete [11926.982491] LustreError: 390037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112364 with bad export cookie 4837715883519806962 [11927.000509] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11937.508750] Lustre: server umount lustre-MDT0001 complete [11947.886144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.687206] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11956.761524] LustreError: 394502:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11962.965690] Lustre: 394841:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.150@tcp [11965.097659] Lustre: server umount MGS complete [11976.677986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11977.156524] LustreError: 395566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11977.184131] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11977.189242] Lustre: Skipped 1 previous similar message [11977.260200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11982.309259] LustreError: 395567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.795399] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11992.559812] LustreError: 395567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.582123] LustreError: 395567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11993.750913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11993.995232] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11994.105640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11998.938937] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12006.906725] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12013.202973] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12015.239078] Lustre: Permanently reactivating lustre-OST0000 [12015.275832] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12015.279581] Lustre: Skipped 2 previous similar messages [12043.233377] LustreError: 397347:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12052.794735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12053.200945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12059.984758] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12068.165872] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12073.961408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.964802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12073.966376] Lustre: Skipped 1 previous similar message [12073.970044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12073.981934] Lustre: Skipped 1 previous similar message [12075.292740] Lustre: server umount lustre-OST0000 complete [12081.124310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12081.141333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12084.195532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12084.211305] Lustre: Skipped 1 previous similar message [12085.935819] Lustre: server umount lustre-MDT0000 complete [12089.314475] LustreError: 395561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.346083] LustreError: 395561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12090.658548] LustreError: 399179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112527 with bad export cookie 4837715883519808019 [12090.666369] LustreError: 399179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12090.669759] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12091.387588] Lustre: server umount lustre-MDT0001 complete [12107.981742] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12112.231657] Key type lgssc unregistered [12112.643041] LNet: 399935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12112.657098] LNetError: 399935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12112.670416] LNet: Removed LNI 192.168.202.150@tcp [12113.633872] Key type .llcrypt unregistered [12113.635344] Key type ._llcrypt unregistered [12133.811958] Key type ._llcrypt registered [12133.813055] Key type .llcrypt registered [12133.924731] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [12152.896523] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12154.008532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12154.058709] alg: No test for adler32 (adler32-zlib) [12155.349936] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12155.732586] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12157.389380] Key type lgssc registered [12158.457886] Lustre: Echo OBD driver; http://www.lustre.org/ [12191.600829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12191.616073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12192.872745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12192.904173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12192.997923] Lustre: lustre-MDT0000: new disk, initializing [12193.105204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12193.127697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12198.005402] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12208.182912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.309710] Lustre: 403093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12208.361739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12208.365458] Lustre: Skipped 1 previous similar message [12208.435685] Lustre: lustre-MDT0001: new disk, initializing [12208.514409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12208.567206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12208.575989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12212.872724] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12221.149412] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12226.254444] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12232.686261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12233.102275] Lustre: lustre-OST0000: new disk, initializing [12233.107681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12233.116844] Lustre: 405061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12233.235158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12238.395218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12238.408928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12238.545108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12240.115633] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12248.236594] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12252.415658] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12252.578914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12256.198682] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12256.416525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12259.302862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.309933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12259.320410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12264.398796] Lustre: server umount lustre-OST0000 complete [12269.538887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.541145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12269.550982] Lustre: Skipped 2 previous similar messages [12269.567847] Lustre: Skipped 2 previous similar messages [12274.657099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12274.668365] Lustre: Skipped 1 previous similar message [12279.781292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12279.789269] Lustre: Skipped 1 previous similar message [12282.335162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12282.462410] Lustre: server umount lustre-MDT0000 complete [12286.085272] LustreError: 403086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112723 with bad export cookie 16833807876444349951 [12286.094106] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12286.096358] LustreError: 403086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12286.419464] Lustre: server umount lustre-MDT0001 complete [12293.124398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12294.661672] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:25:31 (1786112731) [12300.707997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12301.131612] LustreError: 407521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.149842] LustreError: 407521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12301.198819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.850250] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12306.405840] LustreError: 407522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.529826] LustreError: 407521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.782547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12313.285287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12317.967188] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12324.518555] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12329.039584] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12334.212816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12334.429930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12338.571351] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12343.758359] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12345.477944] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12351.875215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12357.092446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12357.099595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12357.103562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12357.108279] Lustre: Skipped 1 previous similar message [12357.125301] Lustre: Skipped 1 previous similar message [12362.219455] LustreError: 409345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12362.238435] LustreError: 409345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12362.287275] Lustre: server umount lustre-OST0000 complete [12367.330179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12367.332790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12367.348630] Lustre: Skipped 1 previous similar message [12367.353645] Lustre: Skipped 1 previous similar message [12371.447685] Lustre: server umount lustre-MDT0000 complete [12372.451172] LustreError: 410156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12372.467568] LustreError: 410156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12373.918699] LustreError: 407502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112811 with bad export cookie 16833807876444350840 [12373.926488] LustreError: 407502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12373.929702] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12374.172493] Lustre: server umount lustre-MDT0001 complete [12380.939332] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12382.536109] Key type lgssc unregistered [12382.738606] LNet: 411248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12382.743048] LNetError: 411248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12382.754702] LNet: Removed LNI 192.168.202.150@tcp [12383.216699] Key type .llcrypt unregistered [12383.219366] Key type ._llcrypt unregistered [12396.190861] Key type ._llcrypt registered [12396.192504] Key type .llcrypt registered [12396.244246] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12396.450337] Key type .llcrypt unregistered [12396.452675] Key type ._llcrypt unregistered [12407.970914] Key type ._llcrypt registered [12407.972899] Key type .llcrypt registered [12408.060217] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12408.696868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12408.712840] alg: No test for adler32 (adler32-zlib) [12409.664623] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12409.797495] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12411.415149] Key type lgssc registered [12412.053024] Lustre: Echo OBD driver; http://www.lustre.org/ [12417.170418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12417.185156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12418.431541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12421.007773] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12425.678441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12425.879141] LustreError: 413442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.888877] LustreError: 413442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12425.893420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12427.938312] LustreError: 413443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.949290] LustreError: 413443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12427.963408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12429.247354] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12433.378579] LustreError: 413438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.396984] LustreError: 413438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12433.710623] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12437.576104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12437.726564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12439.651306] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12453.345547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12453.352948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.358660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12455.803526] Lustre: server umount lustre-OST0000 complete [12458.464855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12458.471231] Lustre: Skipped 1 previous similar message [12458.475249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12458.478424] Lustre: Skipped 2 previous similar messages [12460.289127] Lustre: server umount lustre-MDT0000 complete [12462.498474] LustreError: 415638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786112899 with bad export cookie 18299164920655462132 [12462.499776] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12462.502744] LustreError: 415638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12462.696441] Lustre: server umount lustre-MDT0001 complete [12469.096831] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12470.660281] Key type lgssc unregistered [12470.855272] LNet: 416593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12470.858406] LNetError: 416593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12470.868888] LNet: Removed LNI 192.168.202.150@tcp [12471.312798] Key type .llcrypt unregistered [12471.314155] Key type ._llcrypt unregistered [12485.182668] Key type ._llcrypt registered [12485.184280] Key type .llcrypt registered [12485.239069] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12485.856567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12485.877675] alg: No test for adler32 (adler32-zlib) [12486.766113] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12486.870909] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12488.463203] Key type lgssc registered [12489.015647] Lustre: Echo OBD driver; http://www.lustre.org/ [12493.079670] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12493.089304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12508.575402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12513.695649] LustreError: 417977:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [12540.319178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12542.640391] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12545.597680] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12548.617547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12548.815340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12548.850236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12550.575491] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12554.210434] LustreError: 418002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12554.221129] LustreError: 418002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12554.330832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12554.499637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12556.280543] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12559.443864] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12561.941909] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12565.985487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.987958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12565.991352] Lustre: Skipped 1 previous similar message [12571.114127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12571.118432] Lustre: Skipped 1 previous similar message [12572.086401] Lustre: server umount lustre-OST0000 complete [12575.200563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12575.208292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12575.222819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12576.229972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12576.236082] LustreError: 419016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.268891] Lustre: server umount lustre-MDT0000 complete [12577.970239] LustreError: 417984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113015 with bad export cookie 2832430682915371459 [12577.973455] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12577.973665] LustreError: 417984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12578.105391] Lustre: server umount lustre-MDT0001 complete [12582.911344] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12584.198643] Key type lgssc unregistered [12584.362434] LNet: 421543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12584.365961] LNetError: 421543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12584.375705] LNet: Removed LNI 192.168.202.150@tcp [12584.743549] Key type .llcrypt unregistered [12584.745118] Key type ._llcrypt unregistered [12595.002531] Key type ._llcrypt registered [12595.003627] Key type .llcrypt registered [12595.049365] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12595.519910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12595.564384] alg: No test for adler32 (adler32-zlib) [12596.427699] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12596.524947] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12598.103169] Key type lgssc registered [12598.549772] Lustre: Echo OBD driver; http://www.lustre.org/ [12601.576859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12601.586102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12602.766680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12604.573583] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12608.048057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12608.235841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12609.911553] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12612.737324] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12614.859064] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12617.514281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12617.639989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12619.874419] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12622.828891] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12627.879315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12633.059154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12633.060479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12633.064141] Lustre: Skipped 2 previous similar messages [12633.066591] Lustre: Skipped 1 previous similar message [12634.246687] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [12634.250357] Lustre: Skipped 1 previous similar message [12635.994394] Lustre: server umount lustre-MDT0000 complete [12636.860515] LustreError: 422917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.50@tcp arrived at 1786113073 with bad export cookie 2062388645952919473 [12636.865755] LustreError: 422917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12637.602310] LustreError: 422919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113074 with bad export cookie 2062388645952918850 [12637.607752] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12638.176544] LustreError: 422933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.177310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.188318] LustreError: 422933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12638.193089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12642.208112] LustreError: 422936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.208379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12642.215397] LustreError: 422936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12643.736651] Lustre: server umount lustre-OST0000 complete [12646.418255] Lustre: server umount lustre-MDT0001 complete [12650.490470] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12651.577086] Key type lgssc unregistered [12651.704392] LNet: 426625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12651.706838] LNetError: 426625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12651.713279] LNet: Removed LNI 192.168.202.150@tcp [12652.037513] Key type .llcrypt unregistered [12652.038774] Key type ._llcrypt unregistered [12662.165147] Key type ._llcrypt registered [12662.166130] Key type .llcrypt registered [12662.205089] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12662.611572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12662.635428] alg: No test for adler32 (adler32-zlib) [12663.499284] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12663.601044] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12665.191159] Key type lgssc registered [12665.569126] Lustre: Echo OBD driver; http://www.lustre.org/ [12668.249094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12668.256654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12669.449160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12670.972363] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12674.321483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12674.498117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12676.221254] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12679.455915] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12681.804701] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12684.852160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12685.009835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12687.623647] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12690.594290] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12694.246547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12699.617857] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12699.623292] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12699.625517] Lustre: Skipped 1 previous similar message [12702.328878] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (stopping) [12704.087883] Lustre: server umount lustre-MDT0001 complete [12704.736905] LustreError: 428017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.743608] LustreError: 428017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12709.857240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12709.857306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.857522] LustreError: 428754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.872951] Lustre: Skipped 1 previous similar message [12709.875543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.963799] Lustre: server umount lustre-OST0000 complete [12713.565933] Lustre: server umount lustre-MDT0000 complete [12718.603532] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12719.722907] Key type lgssc unregistered [12719.864453] LNet: 431700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12719.867105] LNetError: 431700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12719.876311] LNet: Removed LNI 192.168.202.150@tcp [12720.206522] Key type .llcrypt unregistered [12720.207970] Key type ._llcrypt unregistered [12730.012708] Key type ._llcrypt registered [12730.013904] Key type .llcrypt registered [12730.052996] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [12736.950333] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12737.395993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12737.418255] alg: No test for adler32 (adler32-zlib) [12738.296118] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12738.401972] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12739.991133] Key type lgssc registered [12740.406913] Lustre: Echo OBD driver; http://www.lustre.org/ [12754.393888] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12761.068735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12761.076407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12762.175614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12762.188072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12762.226331] Lustre: lustre-MDT0000: new disk, initializing [12762.254231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12762.260851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12763.872320] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12770.258843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12770.359881] Lustre: lustre-OST0000: new disk, initializing [12770.362536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12770.364935] Lustre: Skipped 1 previous similar message [12770.368600] Lustre: 436704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12770.397938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12772.785248] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12773.653803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12773.658210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12773.672121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12777.359552] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12784.006966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12784.054732] Lustre: 435687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12784.082940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12784.099931] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (not set up) [12784.129115] Lustre: lustre-MDT0001: new disk, initializing [12784.178631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12784.195822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12784.199470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12785.729338] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12790.002065] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12790.087400] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12791.632089] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12794.783870] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [12798.946926] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12798.948122] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12798.948694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12798.951303] Lustre: Skipped 1 previous similar message [12802.715213] Lustre: server umount lustre-MDT0001 complete [12809.187852] LustreError: 439296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12809.199356] LustreError: 439296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12814.305972] LustreError: 435694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.311600] LustreError: 435694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12819.424527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12819.424554] LustreError: 438044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12819.428393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12819.436014] Lustre: Skipped 2 previous similar messages [12821.521378] Lustre: server umount lustre-OST0000 complete [12829.463919] Lustre: server umount lustre-MDT0000 complete [12834.686372] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12835.720921] Key type lgssc unregistered [12835.856336] LNet: 440472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12835.858838] LNetError: 440472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12835.867858] LNet: Removed LNI 192.168.202.150@tcp [12836.227113] Key type .llcrypt unregistered [12836.228742] Key type ._llcrypt unregistered [12850.871219] Key type ._llcrypt registered [12850.872422] Key type .llcrypt registered [12850.910092] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12851.028733] Key type .llcrypt unregistered [12851.029877] Key type ._llcrypt unregistered [12858.033332] Key type ._llcrypt registered [12858.034368] Key type .llcrypt registered [12858.075028] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12858.417573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12858.475275] alg: No test for adler32 (adler32-zlib) [12859.331424] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [12859.423216] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [12861.007088] Key type lgssc registered [12861.385288] Lustre: Echo OBD driver; http://www.lustre.org/ [12870.282624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12870.287949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12871.364778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12871.374305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12871.412605] Lustre: lustre-MDT0000: new disk, initializing [12871.438877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12871.445835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12872.869924] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12877.158045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12877.203127] Lustre: 443824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12877.228503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12877.231445] Lustre: Skipped 1 previous similar message [12877.265991] Lustre: lustre-MDT0001: new disk, initializing [12877.301639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12877.313911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12877.317773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12878.719365] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12883.099677] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12885.600060] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12888.301361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12888.412590] Lustre: lustre-OST0000: new disk, initializing [12888.414818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12888.418298] Lustre: 445793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12888.448721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12890.683373] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12894.712334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12894.715802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12894.751894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12895.060164] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12897.760040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12897.762943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12897.768249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12899.813348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.813481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12899.820880] Lustre: Skipped 1 previous similar message [12899.823143] Lustre: Skipped 1 previous similar message [12903.791036] Lustre: server umount lustre-MDT0000 complete [12905.181257] LustreError: 443816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113342 with bad export cookie 9051006903999823307 [12905.184909] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12905.186357] LustreError: 443816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12905.307846] Lustre: server umount lustre-MDT0001 complete [12916.907410] Lustre: server umount lustre-OST0000 complete [12920.925084] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:35:58 (1786113358) [12923.882570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12924.062165] LustreError: 447818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.069200] LustreError: 447818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12924.089281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12925.550817] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12928.627721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12928.793908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12930.394983] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12933.234503] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12935.146742] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12950.563024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12950.657707] Lustre: Found index 0 for lustre-OST0000, updating log [12950.725710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12952.953749] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12955.752528] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12970.213886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12974.560081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12974.562585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12974.567864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12974.570216] Lustre: Skipped 1 previous similar message [12979.050161] Lustre: server umount lustre-OST0000 complete [12995.552206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12995.554898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12995.558660] Lustre: Skipped 1 previous similar message [12995.561340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12995.563374] Lustre: Skipped 3 previous similar messages [13001.185631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13001.188198] Lustre: Skipped 3 previous similar messages [13006.303219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13006.368757] Lustre: server umount lustre-MDT0000 complete [13007.330696] LustreError: 447818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.337226] LustreError: 447818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13007.789427] LustreError: 448553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786113444 with bad export cookie 9051006903999824686 [13007.793594] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13007.795831] LustreError: 448553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13007.903202] LNet: 1 local NIs in recovery (showing 1): 192.168.202.150@tcp [13007.925896] Lustre: server umount lustre-MDT0001 complete [13011.520577] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:37:28 (1786113448) [13601.520468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13601.760454] LustreError: 454244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13601.806665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13604.575568] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13606.883718] LustreError: 454245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13610.424103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13610.721921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13613.238923] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13617.990180] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13621.461742] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13638.564087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13638.744662] Lustre: Found index 0 for lustre-OST0000, updating log [13638.989918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13642.763400] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13647.445685] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13648.722591] Lustre: server umount lustre-OST0000 complete [13651.936392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13651.938888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13651.943133] Lustre: Skipped 1 previous similar message [13651.945919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13651.948673] Lustre: Skipped 2 previous similar messages [13656.032528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13656.034698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13656.939621] Lustre: server umount lustre-MDT0000 complete [13657.128607] LustreError: 454245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13657.144733] LustreError: 454245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13659.424238] LustreError: 454225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114096 with bad export cookie 9051006903999825757 [13659.428056] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13659.429922] LustreError: 454225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13659.646517] Lustre: server umount lustre-MDT0001 complete [13660.259066] LNet: 1 local NIs in recovery (showing 1): 192.168.202.150@tcp [13681.559445] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [13687.981505] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13712.210961] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:49:09 (1786114149) [13722.270741] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [13723.907509] Key type lgssc unregistered [13724.056889] LNet: 460854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13724.060208] LNetError: 460854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13768.101521] LNet: Removed LNI 192.168.202.150@tcp [13768.592795] Key type .llcrypt unregistered [13768.594966] Key type ._llcrypt unregistered [13780.081436] Key type ._llcrypt registered [13780.084604] Key type .llcrypt registered [13780.158332] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13780.921289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13780.936080] alg: No test for adler32 (adler32-zlib) [13781.864295] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [13782.031889] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [13783.695282] Key type lgssc registered [13784.442628] Lustre: Echo OBD driver; http://www.lustre.org/ [13787.008692] LNet: Added LNI 192.168.202.150@tcp100 [8/256/0/180] [13787.570629] LNet: Added LNI 192.168.202.150@tcp101 [8/256/0/180] [13788.103383] LNet: Added LNI 192.168.202.150@tcp102 [8/256/0/180] [13788.645400] LNet: Added LNI 192.168.202.150@tcp103 [8/256/0/180] [13789.195632] LNet: Added LNI 192.168.202.150@tcp104 [8/256/0/180] [13789.733689] LNet: Added LNI 192.168.202.150@tcp105 [8/256/0/180] [13790.250486] LNet: Added LNI 192.168.202.150@tcp106 [8/256/0/180] [13790.826693] LNet: Added LNI 192.168.202.150@tcp107 [8/256/0/180] [13791.330484] LNet: Added LNI 192.168.202.150@tcp108 [8/256/0/180] [13791.898673] LNet: Added LNI 192.168.202.150@tcp109 [8/256/0/180] [13792.466417] LNet: Added LNI 192.168.202.150@tcp110 [8/256/0/180] [13793.060529] LNet: Added LNI 192.168.202.150@tcp111 [8/256/0/180] [13793.633656] LNet: Added LNI 192.168.202.150@tcp112 [8/256/0/180] [13794.107285] LNet: Added LNI 192.168.202.150@tcp113 [8/256/0/180] [13794.572199] LNet: Added LNI 192.168.202.150@tcp114 [8/256/0/180] [13795.094758] LNet: Added LNI 192.168.202.150@tcp115 [8/256/0/180] [13795.662918] LNet: Added LNI 192.168.202.150@tcp116 [8/256/0/180] [13796.265967] LNet: Added LNI 192.168.202.150@tcp117 [8/256/0/180] [13796.999234] LNet: Added LNI 192.168.202.150@tcp118 [8/256/0/180] [13797.705273] LNet: Added LNI 192.168.202.150@tcp119 [8/256/0/180] [13798.286802] LNet: Added LNI 192.168.202.150@tcp120 [8/256/0/180] [13798.886565] LNet: Added LNI 192.168.202.150@tcp121 [8/256/0/180] [13799.505494] LNet: Added LNI 192.168.202.150@tcp122 [8/256/0/180] [13800.066673] LNet: Added LNI 192.168.202.150@tcp123 [8/256/0/180] [13800.674831] LNet: Added LNI 192.168.202.150@tcp124 [8/256/0/180] [13801.273789] LNet: Added LNI 192.168.202.150@tcp125 [8/256/0/180] [13801.859643] LNet: Added LNI 192.168.202.150@tcp126 [8/256/0/180] [13802.432586] LNet: Added LNI 192.168.202.150@tcp127 [8/256/0/180] [13803.075611] LNet: Added LNI 192.168.202.150@tcp128 [8/256/0/180] [13803.625757] LNet: Added LNI 192.168.202.150@tcp129 [8/256/0/180] [13804.167892] LNet: Added LNI 192.168.202.150@tcp130 [8/256/0/180] [13804.715803] LNet: Added LNI 192.168.202.150@tcp131 [8/256/0/180] [13805.277913] LNet: Added LNI 192.168.202.150@tcp132 [8/256/0/180] [13805.891871] LNet: Added LNI 192.168.202.150@tcp133 [8/256/0/180] [13806.449704] LNet: Added LNI 192.168.202.150@tcp134 [8/256/0/180] [13807.088585] LNet: Added LNI 192.168.202.150@tcp135 [8/256/0/180] [13811.800866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13811.818202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13812.889791] Lustre: 463852:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.150@tcp: can't add NID 192.168.202.150@tcp130: rc = -75 [13812.930184] LNetError: 461314:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.150 [13812.944513] LNetError: Protocol error connecting to 192.168.202.150@tcp121 at host 192.168.202.150:988: is it running a compatible version of Lustre? [13812.958966] LNetError: Skipped 7 previous similar messages [13813.001613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13813.025202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13813.075581] Lustre: lustre-MDT0000: new disk, initializing [13813.137270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13813.151646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13813.792205] LNet: 3 local NIs in recovery (showing 3): 192.168.202.150@tcp129, 192.168.202.150@tcp127, 192.168.202.150@tcp121 [13813.793678] LNetError: 461315:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.150 [13813.802993] LNetError: 461315:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13813.806329] LNetError: Protocol error connecting to 192.168.202.150@tcp106 at host 192.168.202.150:988: is it running a compatible version of Lustre? [13813.810879] LNetError: Skipped 2 previous similar messages [13815.699573] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13822.608275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13822.724167] Lustre: 463868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13822.779917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13822.787521] Lustre: Skipped 1 previous similar message [13822.847133] Lustre: lustre-MDT0001: new disk, initializing [13822.870100] Lustre: 464670:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.150@tcp: can't add NID 192.168.202.150@tcp130: rc = -75 [13822.879831] Lustre: 464670:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13822.976878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13823.001280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13823.008329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13825.493938] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13828.619093] Lustre: 465105:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.150@tcp: can't add NID 192.168.202.150@tcp130: rc = -75 [13828.623022] Lustre: 465105:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13831.364328] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13834.563191] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13838.147626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13838.302298] Lustre: lustre-OST0000: new disk, initializing [13838.304657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13838.309416] Lustre: 465846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13838.348729] Lustre: 465851:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.150@tcp: can't add NID 192.168.202.150@tcp130: rc = -75 [13838.352065] Lustre: 465851:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13838.379089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13841.454323] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13842.464601] LNetError: 463877:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.150 [13842.468367] LNetError: 463877:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13842.470785] LNetError: Protocol error connecting to 192.168.202.150@tcp106 at host 192.168.202.150:988: is it running a compatible version of Lustre? [13844.524936] LNetError: 461312:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.150 [13844.531628] LNetError: Protocol error connecting to 192.168.202.150@tcp122 at host 192.168.202.150:988: is it running a compatible version of Lustre? [13844.990252] Lustre: 466418:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.150@tcp: can't add NID 192.168.202.150@tcp130: rc = -75 [13844.993853] Lustre: 466418:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13845.051182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13845.056760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13845.120553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13846.962403] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13849.636503] LNetError: 463882:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.150 [13849.644448] LNetError: 463882:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13849.649631] LNetError: Protocol error connecting to 192.168.202.150@tcp128 at host 192.168.202.150:988: is it running a compatible version of Lustre? [13849.655520] LNetError: Skipped 1 previous similar message [13855.200640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13855.204544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13855.210068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13855.715828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13855.723659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13855.742768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13857.284179] Lustre: server umount lustre-OST0000 complete [13858.852361] LNetError: 463878:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.150 [13858.862692] LNetError: 463878:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13858.872287] LNetError: Protocol error connecting to 192.168.202.150@tcp125 at host 192.168.202.150:988: is it running a compatible version of Lustre? [13858.881531] LNetError: Skipped 7 previous similar messages [13860.835972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13860.839840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13860.843160] Lustre: Skipped 1 previous similar message [13865.506598] Lustre: server umount lustre-MDT0000 complete [13867.329042] LustreError: 463862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114304 with bad export cookie 6149862598433278519 [13867.332109] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13867.334843] LustreError: 463862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13867.502541] Lustre: server umount lustre-MDT0001 complete [13873.073436] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [13874.514738] Key type lgssc unregistered [13874.706942] LNet: 467758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13874.713436] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13875.745149] LNet: Removed LNI 192.168.202.150@tcp [13875.746908] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13875.751966] LNet: Removed LNI 192.168.202.150@tcp100 [13876.768425] LNet: Removed LNI 192.168.202.150@tcp101 [13876.774142] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13876.777419] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13876.782897] LNet: Removed LNI 192.168.202.150@tcp102 [13877.791153] LNet: Removed LNI 192.168.202.150@tcp103 [13878.815125] LNet: Removed LNI 192.168.202.150@tcp104 [13878.816927] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13878.820156] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13878.823337] LNet: Removed LNI 192.168.202.150@tcp105 [13879.839175] LNet: Removed LNI 192.168.202.150@tcp106 [13880.863209] LNet: Removed LNI 192.168.202.150@tcp107 [13881.887157] LNet: Removed LNI 192.168.202.150@tcp108 [13882.911162] LNet: Removed LNI 192.168.202.150@tcp109 [13882.917037] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13882.921468] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13882.924835] LNet: Removed LNI 192.168.202.150@tcp110 [13883.935143] LNet: Removed LNI 192.168.202.150@tcp111 [13884.959334] LNet: Removed LNI 192.168.202.150@tcp112 [13885.983173] LNet: Removed LNI 192.168.202.150@tcp113 [13887.007122] LNet: Removed LNI 192.168.202.150@tcp114 [13888.031139] LNet: Removed LNI 192.168.202.150@tcp115 [13889.055144] LNet: Removed LNI 192.168.202.150@tcp116 [13890.079810] LNet: Removed LNI 192.168.202.150@tcp117 [13891.103517] LNet: Removed LNI 192.168.202.150@tcp118 [13891.107545] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13891.113634] LNetError: 467758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13891.117454] LNet: Removed LNI 192.168.202.150@tcp119 [13892.127168] LNet: Removed LNI 192.168.202.150@tcp120 [13893.151130] LNet: Removed LNI 192.168.202.150@tcp121 [13894.175151] LNet: Removed LNI 192.168.202.150@tcp122 [13895.199291] LNet: Removed LNI 192.168.202.150@tcp123 [13896.223166] LNet: Removed LNI 192.168.202.150@tcp124 [13897.247178] LNet: Removed LNI 192.168.202.150@tcp125 [13898.271620] LNet: Removed LNI 192.168.202.150@tcp126 [13899.295728] LNet: Removed LNI 192.168.202.150@tcp127 [13900.319646] LNet: Removed LNI 192.168.202.150@tcp128 [13901.343164] LNet: Removed LNI 192.168.202.150@tcp129 [13901.345020] LNet: Removed LNI 192.168.202.150@tcp130 [13901.346572] LNet: Removed LNI 192.168.202.150@tcp131 [13901.348284] LNet: Removed LNI 192.168.202.150@tcp132 [13901.349950] LNet: Removed LNI 192.168.202.150@tcp133 [13901.351502] LNet: Removed LNI 192.168.202.150@tcp134 [13901.359076] LNet: Removed LNI 192.168.202.150@tcp135 [13901.691367] Key type .llcrypt unregistered [13901.692419] Key type ._llcrypt unregistered [13913.206016] Key type ._llcrypt registered [13913.207258] Key type .llcrypt registered [13913.253601] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [13920.411869] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13920.830802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13920.892126] alg: No test for adler32 (adler32-zlib) [13921.782822] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [13921.887654] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [13923.487154] Key type lgssc registered [13924.063950] Lustre: Echo OBD driver; http://www.lustre.org/ [13940.891915] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:52:57 (1786114377) [13949.134755] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [13950.572298] Key type lgssc unregistered [13950.725538] LNet: 471822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13950.728753] LNetError: 471822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13950.737282] LNet: Removed LNI 192.168.202.150@tcp [13951.188120] Key type .llcrypt unregistered [13951.189711] Key type ._llcrypt unregistered [13959.158375] Key type ._llcrypt registered [13959.159554] Key type .llcrypt registered [13959.213721] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13959.805190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13959.825077] alg: No test for adler32 (adler32-zlib) [13960.731334] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [13960.860613] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [13962.487764] Key type lgssc registered [13963.052209] Lustre: Echo OBD driver; http://www.lustre.org/ [13967.704378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13967.712174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13968.854652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13968.873550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13968.925602] Lustre: lustre-MDT0000: new disk, initializing [13968.966246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13968.980238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13971.190281] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13977.291083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13977.361658] Lustre: 473013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13977.410160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13977.411682] Lustre: Skipped 1 previous similar message [13977.465722] Lustre: lustre-MDT0001: new disk, initializing [13977.520030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13977.550077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13977.554102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13979.878510] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13984.840336] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13987.377396] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13991.019453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13991.169174] Lustre: lustre-OST0000: new disk, initializing [13991.170982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13991.178571] Lustre: 474979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13991.234068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13993.289781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13993.294654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13993.369387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13993.990688] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13999.396876] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14001.193822] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing lnet_if_list [14004.075283] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14004.108182] LNet: Added LNI 192.168.202.150@tcp10 [8/256/0/180] [14006.978655] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14007.004863] LNet: Added LNI 192.168.202.150@tcp11 [8/256/0/180] [14008.923016] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14008.943029] LNet: Added LNI 192.168.202.150@tcp12 [8/256/0/180] [14011.012858] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14011.029178] LNet: Added LNI 192.168.202.150@tcp13 [8/256/0/180] [14013.492209] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14013.518820] LNet: Added LNI 192.168.202.150@tcp14 [8/256/0/180] [14015.897184] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14015.915820] LNet: Added LNI 192.168.202.150@tcp15 [8/256/0/180] [14018.462217] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14018.485619] LNet: Added LNI 192.168.202.150@tcp16 [8/256/0/180] [14020.725664] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14020.741648] LNet: Added LNI 192.168.202.150@tcp17 [8/256/0/180] [14023.243672] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14023.258341] LNet: Added LNI 192.168.202.150@tcp18 [8/256/0/180] [14025.592113] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14025.614463] LNet: Added LNI 192.168.202.150@tcp19 [8/256/0/180] [14027.911597] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14027.925848] LNet: Added LNI 192.168.202.150@tcp20 [8/256/0/180] [14030.382039] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14030.397796] LNet: Added LNI 192.168.202.150@tcp21 [8/256/0/180] [14032.910381] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14032.931266] LNet: Added LNI 192.168.202.150@tcp22 [8/256/0/180] [14035.271571] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14035.288182] LNet: Added LNI 192.168.202.150@tcp23 [8/256/0/180] [14037.860646] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14037.875592] LNet: Added LNI 192.168.202.150@tcp24 [8/256/0/180] [14040.406996] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14040.424442] LNet: Added LNI 192.168.202.150@tcp25 [8/256/0/180] [14042.975455] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14042.990298] LNet: Added LNI 192.168.202.150@tcp26 [8/256/0/180] [14045.524192] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14045.536059] LNet: Added LNI 192.168.202.150@tcp27 [8/256/0/180] [14048.181463] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14048.195470] LNet: Added LNI 192.168.202.150@tcp28 [8/256/0/180] [14051.014750] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14051.030503] LNet: Added LNI 192.168.202.150@tcp29 [8/256/0/180] [14054.229526] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14054.247732] LNet: Added LNI 192.168.202.150@tcp30 [8/256/0/180] [14057.227940] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14057.245372] LNet: Added LNI 192.168.202.150@tcp31 [8/256/0/180] [14060.606553] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14060.619126] LNet: Added LNI 192.168.202.150@tcp32 [8/256/0/180] [14063.716244] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14063.736620] LNet: Added LNI 192.168.202.150@tcp33 [8/256/0/180] [14066.573582] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14066.589621] LNet: Added LNI 192.168.202.150@tcp34 [8/256/0/180] [14069.685440] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14069.697205] LNet: Added LNI 192.168.202.150@tcp35 [8/256/0/180] [14072.748719] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14072.765322] LNet: Added LNI 192.168.202.150@tcp36 [8/256/0/180] [14075.722041] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14075.736249] LNet: Added LNI 192.168.202.150@tcp37 [8/256/0/180] [14078.765332] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14078.786498] LNet: Added LNI 192.168.202.150@tcp38 [8/256/0/180] [14082.107364] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14082.121712] LNet: Added LNI 192.168.202.150@tcp39 [8/256/0/180] [14085.574750] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14085.594411] LNet: Added LNI 192.168.202.150@tcp40 [8/256/0/180] [14089.060315] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14089.083381] LNet: Added LNI 192.168.202.150@tcp41 [8/256/0/180] [14092.308576] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14092.323956] LNet: Added LNI 192.168.202.150@tcp42 [8/256/0/180] [14095.718907] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14095.734482] LNet: Added LNI 192.168.202.150@tcp43 [8/256/0/180] [14099.147262] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14099.163800] LNet: Added LNI 192.168.202.150@tcp44 [8/256/0/180] [14102.645460] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14102.664264] LNet: Added LNI 192.168.202.150@tcp45 [8/256/0/180] [14105.956380] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14105.975517] LNet: Added LNI 192.168.202.150@tcp46 [8/256/0/180] [14109.370912] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14109.385739] LNet: Added LNI 192.168.202.150@tcp47 [8/256/0/180] [14112.678320] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14112.693351] LNet: Added LNI 192.168.202.150@tcp48 [8/256/0/180] [14116.399643] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14116.414888] LNet: Added LNI 192.168.202.150@tcp49 [8/256/0/180] [14120.074844] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14120.088800] LNet: Added LNI 192.168.202.150@tcp50 [8/256/0/180] [14131.169425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14131.172393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14131.178139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14131.682417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14131.684681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14131.689710] Lustre: Skipped 1 previous similar message [14131.694234] Lustre: Skipped 1 previous similar message [14135.599505] Lustre: server umount lustre-MDT0000 complete [14137.297370] LustreError: 473807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114574 with bad export cookie 5959077442559116557 [14137.299610] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14137.301892] LustreError: 473807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14137.423871] Lustre: server umount lustre-MDT0001 complete [14146.482287] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [14150.131510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14158.306336] Lustre: 472285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114579/real 1786114579] req@00000000a71aaaae x1872876690048512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786114595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14158.335726] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14159.327169] Lustre: 472286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114580/real 1786114580] req@00000000db3d8557 x1872876690048640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786114596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14162.464924] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x52b2e55c88c78dcd [14162.472889] Lustre: MGC192.168.202.150@tcp: Connection restored to 0@lo (at 0@lo) [14162.612147] LustreError: 474972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14162.621490] LustreError: 474972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14162.654277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14163.424070] Lustre: 472288:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114584/real 1786114584] req@0000000088f6b4fb x1872876690049024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786114600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14163.682419] LustreError: 474973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14164.852647] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14167.712939] LustreError: 474972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14168.976739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14169.045294] Lustre: 472286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114589/real 1786114589] req@000000001f03620f x1872876690049152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786114605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14169.065496] Lustre: 472286:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14169.186380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14171.291916] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14173.283090] Lustre: 472285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786114594/real 1786114594] req@00000000ea71d2cb x1872876690049280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786114610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14174.695933] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14175.021155] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14177.950341] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14186.985228] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14188.512534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14188.516772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14188.527889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14189.537923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14192.101714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14192.109590] Lustre: Skipped 2 previous similar messages [14193.832480] Lustre: server umount lustre-OST0000 complete [14197.217728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14197.218721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14197.222412] Lustre: Skipped 2 previous similar messages [14201.756461] Lustre: server umount lustre-MDT0000 complete [14202.337284] LustreError: 499175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14202.346607] LustreError: 499175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14203.200493] LustreError: 474978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114640 with bad export cookie 5959077442559118797 [14203.203690] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14203.205390] LustreError: 474978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14203.370970] Lustre: server umount lustre-MDT0001 complete [14207.870375] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [14209.005879] Key type lgssc unregistered [14209.173634] LNet: 501954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14209.177923] LNetError: 501954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14209.181143] LNet: Removed LNI 192.168.202.150@tcp [14209.183283] LNet: Removed LNI 192.168.202.150@tcp10 [14209.184969] LNet: Removed LNI 192.168.202.150@tcp11 [14209.186316] LNet: Removed LNI 192.168.202.150@tcp12 [14209.188487] LNet: Removed LNI 192.168.202.150@tcp13 [14209.190080] LNet: Removed LNI 192.168.202.150@tcp14 [14209.191743] LNet: Removed LNI 192.168.202.150@tcp15 [14209.193248] LNet: Removed LNI 192.168.202.150@tcp16 [14209.194566] LNet: Removed LNI 192.168.202.150@tcp17 [14209.196278] LNet: Removed LNI 192.168.202.150@tcp18 [14209.198133] LNet: Removed LNI 192.168.202.150@tcp19 [14209.199308] LNet: Removed LNI 192.168.202.150@tcp20 [14209.200975] LNet: Removed LNI 192.168.202.150@tcp21 [14209.202808] LNet: Removed LNI 192.168.202.150@tcp22 [14209.204706] LNet: Removed LNI 192.168.202.150@tcp23 [14209.206332] LNet: Removed LNI 192.168.202.150@tcp24 [14209.208159] LNet: Removed LNI 192.168.202.150@tcp25 [14209.209985] LNet: Removed LNI 192.168.202.150@tcp26 [14209.211751] LNet: Removed LNI 192.168.202.150@tcp27 [14209.213510] LNet: Removed LNI 192.168.202.150@tcp28 [14209.215415] LNet: Removed LNI 192.168.202.150@tcp29 [14209.217467] LNet: Removed LNI 192.168.202.150@tcp30 [14209.219439] LNet: Removed LNI 192.168.202.150@tcp31 [14209.221586] LNet: Removed LNI 192.168.202.150@tcp32 [14209.224153] LNet: Removed LNI 192.168.202.150@tcp33 [14209.225846] LNet: Removed LNI 192.168.202.150@tcp34 [14209.227481] LNet: Removed LNI 192.168.202.150@tcp35 [14209.229201] LNet: Removed LNI 192.168.202.150@tcp36 [14209.230931] LNet: Removed LNI 192.168.202.150@tcp37 [14209.232824] LNet: Removed LNI 192.168.202.150@tcp38 [14209.235077] LNet: Removed LNI 192.168.202.150@tcp39 [14209.236938] LNet: Removed LNI 192.168.202.150@tcp40 [14209.238835] LNet: Removed LNI 192.168.202.150@tcp41 [14209.240737] LNet: Removed LNI 192.168.202.150@tcp42 [14209.242594] LNet: Removed LNI 192.168.202.150@tcp43 [14209.244527] LNet: Removed LNI 192.168.202.150@tcp44 [14209.246234] LNet: Removed LNI 192.168.202.150@tcp45 [14209.248384] LNet: Removed LNI 192.168.202.150@tcp46 [14209.250079] LNet: Removed LNI 192.168.202.150@tcp47 [14209.251538] LNet: Removed LNI 192.168.202.150@tcp48 [14209.252895] LNet: Removed LNI 192.168.202.150@tcp49 [14209.260410] LNet: Removed LNI 192.168.202.150@tcp50 [14209.581306] Key type .llcrypt unregistered [14209.582474] Key type ._llcrypt unregistered [14217.516131] Key type ._llcrypt registered [14217.517430] Key type .llcrypt registered [14217.559086] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [14224.949702] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [14225.398605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14225.416259] alg: No test for adler32 (adler32-zlib) [14226.314982] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [14226.433406] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [14228.039340] Key type lgssc registered [14228.513280] Lustre: Echo OBD driver; http://www.lustre.org/ [14242.947751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14242.961303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14244.076187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14244.091199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14244.133259] Lustre: lustre-MDT0000: new disk, initializing [14244.170136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14244.178090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14245.889719] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14251.017711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14251.070771] Lustre: 505128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14251.097708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14251.099919] Lustre: Skipped 1 previous similar message [14251.139096] Lustre: lustre-MDT0001: new disk, initializing [14251.186043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14251.205645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14251.209725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14252.836585] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14257.468819] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14260.137964] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14263.429427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14263.511600] Lustre: *** cfs_fail_loc=90f, val=0*** [14263.548544] Lustre: lustre-OST0000: new disk, initializing [14263.550851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14263.554214] Lustre: 507142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14263.592757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14266.200461] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14270.638385] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14272.007447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14272.011985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14272.062374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14277.089295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14277.090377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14277.097685] LustreError: Skipped 1 previous similar message [14277.103293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14277.107310] Lustre: Skipped 1 previous similar message [14279.638684] Lustre: server umount lustre-OST0000 complete [14282.208913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14282.209366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14282.209754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14282.214704] Lustre: Skipped 2 previous similar messages [14287.332870] Lustre: server umount lustre-MDT0000 complete [14288.836436] LustreError: 505120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114725 with bad export cookie 14331877957573660274 [14288.839153] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14288.842281] LustreError: 505120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14288.976509] Lustre: server umount lustre-MDT0001 complete [14293.264814] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [14294.480505] Key type lgssc unregistered [14294.618447] LNet: 509048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14294.620978] LNetError: 509048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14294.632333] LNet: Removed LNI 192.168.202.150@tcp [14294.984145] Key type .llcrypt unregistered [14294.985240] Key type ._llcrypt unregistered [14304.228303] Key type ._llcrypt registered [14304.229750] Key type .llcrypt registered [14304.281722] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [14304.751138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14304.759079] alg: No test for adler32 (adler32-zlib) [14305.650786] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [14305.784897] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [14307.375593] Key type lgssc registered [14307.847840] Lustre: Echo OBD driver; http://www.lustre.org/ [14310.791832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14310.800980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14311.960882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14313.557414] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14316.742199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14316.888216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14318.532722] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14321.287075] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14323.487314] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14326.434956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14326.576123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14328.920682] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14331.758902] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14347.620780] loop0: detected capacity change from 0 to 204800000 [14347.775304] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14347.780213] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14347.789936] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14350.759103] loop0: detected capacity change from 0 to 204800000 [14350.835265] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14350.840329] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14350.846191] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14352.941298] loop0: detected capacity change from 0 to 204800000 [14355.132445] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14355.193512] Lustre: fs15246-OST0000: new disk, initializing [14355.195478] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14355.198286] Lustre: 513909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14355.225865] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14357.615942] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14362.330822] loop1: detected capacity change from 0 to 204800000 [14364.648068] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14364.690646] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14364.726438] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14364.767360] Lustre: fs15246-MDT0000: new disk, initializing [14364.830958] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14364.842479] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14364.886383] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14365.920502] Lustre: 515031:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000013c9abb5 x1872877051741056/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786114819 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14366.565908] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14375.013122] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14375.016450] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14375.031476] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14378.979992] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14378.991919] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14381.979383] Lustre: server umount fs15246-MDT0000 complete [14384.102825] LustreError: 512783:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14384.110252] LustreError: 512783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14386.001421] Lustre: server umount fs15246-OST0000 complete [14392.287909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14392.291149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14392.296434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14394.337203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14394.337834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14394.338401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14394.346823] Lustre: Skipped 1 previous similar message [14396.502101] Lustre: server umount lustre-OST0000 complete [14398.944053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14398.946691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14398.951772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14404.576139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14404.579649] Lustre: Skipped 3 previous similar messages [14412.255125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14412.326803] Lustre: server umount lustre-MDT0000 complete [14413.893220] LustreError: 515025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114851 with bad export cookie 8932787677879252364 [14413.897157] LustreError: 515025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14413.897172] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14414.040338] Lustre: server umount lustre-MDT0001 complete [14418.492600] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [14419.765798] Key type lgssc unregistered [14419.933060] LNet: 518125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14419.942497] LNetError: 518125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14419.955848] LNet: Removed LNI 192.168.202.150@tcp [14420.361437] Key type .llcrypt unregistered [14420.365677] Key type ._llcrypt unregistered [14429.702207] Key type ._llcrypt registered [14429.703316] Key type .llcrypt registered [14429.765691] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [14430.131363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14430.137925] alg: No test for adler32 (adler32-zlib) [14431.025286] Lustre: Lustre: Build Version: 2.17.56_50_gf5ff674 [14431.123220] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [14432.711849] Key type lgssc registered [14433.192034] Lustre: Echo OBD driver; http://www.lustre.org/ [14450.089819] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [14454.506015] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [14469.547513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14469.553395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14470.642463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14470.653081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14470.690258] Lustre: lustre-MDT0000: new disk, initializing [14470.718115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14470.726638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14472.424317] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14477.385460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14477.445803] Lustre: 522588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14477.493213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14477.498183] Lustre: Skipped 1 previous similar message [14477.565228] Lustre: lustre-MDT0001: new disk, initializing [14477.608937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14477.630910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14477.636229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14479.267122] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14483.789842] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14486.236660] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14489.383815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14489.545668] Lustre: lustre-OST0000: new disk, initializing [14489.548329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14489.551925] Lustre: 524553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14489.600725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14491.662987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14491.668446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14491.745291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14492.464823] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [14497.254433] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14499.056577] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14499.152454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14500.985606] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14501.081473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14506.976314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14506.978362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14506.983062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14506.986014] Lustre: Skipped 1 previous similar message [14506.993091] Lustre: Skipped 1 previous similar message [14508.155097] Lustre: server umount lustre-OST0000 complete [14512.098638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14512.102083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14512.106568] Lustre: Skipped 1 previous similar message [14517.217436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14517.224244] Lustre: Skipped 2 previous similar messages [14522.340462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14524.383110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14524.482788] Lustre: server umount lustre-MDT0000 complete [14526.235948] LustreError: 522581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786114963 with bad export cookie 1848278965689410058 [14526.237529] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14526.240712] LustreError: 522581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14526.375975] Lustre: server umount lustre-MDT0001 complete [14538.056988] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [14542.693305] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [14557.420602] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14251 sec ========= 11:03:14 (1786114994) [14558.146217] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:03:15 (1786114995) === [14559.777840] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:03:16 (1786114996) === [14578.921529] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [14580.225026] Key type lgssc unregistered [14580.383327] LNet: 531188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14580.386590] LNetError: 531188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14580.396334] LNet: Removed LNI 192.168.202.150@tcp [14580.767975] Key type .llcrypt unregistered [14580.769433] Key type ._llcrypt unregistered