[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496520072 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007367] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011100] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022123] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026705] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028045] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] *** VALIDATE tmpfs *** [ 0.031480] *** VALIDATE proc *** [ 0.032250] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034352] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041965] debug: unmapping init [mem 0xffffffffa8e59000-0xffffffffa8e60fff] [ 0.043913] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.053230] rcu: Hierarchical SRCU implementation. [ 0.055486] smp: Bringing up secondary CPUs ... [ 0.056619] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060249] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136681] node 0 deferred pages initialised in 71ms [ 0.140608] devtmpfs: initialized [ 0.141223] x86/mm: Memory block size: 128MB [ 0.143823] gcov: version magic: 0x41383552 [ 0.144623] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149218] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152303] pinctrl core: initialized pinctrl subsystem [ 0.154163] [ 0.154816] ************************************************************* [ 0.158011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160011] ** ** [ 0.162009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164013] ** ** [ 0.166011] ** This means that this kernel is built to expose internal ** [ 0.168011] ** IOMMU data structures, which may compromise security on ** [ 0.171025] ** your system. ** [ 0.173008] ** ** [ 0.175012] ** If you see this message and you are not debugging the ** [ 0.178014] ** kernel, report this immediately to your vendor! ** [ 0.180010] ** ** [ 0.183011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184008] ************************************************************* [ 0.187769] NET: Registered protocol family 16 [ 0.190412] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199397] cpuidle: using governor menu [ 0.200559] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204020] PCI: Using configuration type 1 for base access [ 0.205121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222089] cryptd: max_cpu_qlen set to 1000 [ 0.226244] ACPI: Added _OSI(Module Device) [ 0.228015] ACPI: Added _OSI(Processor Device) [ 0.230018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237136] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243614] ACPI: Interpreter enabled [ 0.246066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247013] ACPI: Using IOAPIC for interrupt routing [ 0.249093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264433] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280349] acpiphp: Slot [2] registered [ 0.282150] acpiphp: Slot [5] registered [ 0.284139] acpiphp: Slot [6] registered [ 0.286125] acpiphp: Slot [7] registered [ 0.287122] acpiphp: Slot [8] registered [ 0.289126] acpiphp: Slot [9] registered [ 0.291147] acpiphp: Slot [10] registered [ 0.293143] acpiphp: Slot [3] registered [ 0.294116] acpiphp: Slot [4] registered [ 0.296158] acpiphp: Slot [11] registered [ 0.298143] acpiphp: Slot [12] registered [ 0.300117] acpiphp: Slot [13] registered [ 0.302132] acpiphp: Slot [14] registered [ 0.303145] acpiphp: Slot [15] registered [ 0.305111] acpiphp: Slot [16] registered [ 0.307135] acpiphp: Slot [17] registered [ 0.308164] acpiphp: Slot [18] registered [ 0.310108] acpiphp: Slot [19] registered [ 0.312146] acpiphp: Slot [20] registered [ 0.314125] acpiphp: Slot [21] registered [ 0.315094] acpiphp: Slot [22] registered [ 0.317165] acpiphp: Slot [23] registered [ 0.319161] acpiphp: Slot [24] registered [ 0.321218] acpiphp: Slot [25] registered [ 0.323166] acpiphp: Slot [26] registered [ 0.325210] acpiphp: Slot [27] registered [ 0.326142] acpiphp: Slot [28] registered [ 0.328160] acpiphp: Slot [29] registered [ 0.330227] acpiphp: Slot [30] registered [ 0.332260] acpiphp: Slot [31] registered [ 0.334093] PCI host bridge to bus 0000:00 [ 0.336035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.339037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341050] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345055] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359195] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362323] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376034] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.381062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394388] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398075] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.403065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.406845] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.413024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.433025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440169] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485465] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503034] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.549203] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.555017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.561014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.575016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.585028] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.590017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.596016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.623496] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.632021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.642023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.670027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682981] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718040] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.736565] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.741746] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.745460] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.748456] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.751295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.756162] iommu: Default domain type: Passthrough [ 0.758605] SCSI subsystem initialized [ 0.760188] ACPI: bus type USB registered [ 0.762185] usbcore: registered new interface driver usbfs [ 0.764156] usbcore: registered new interface driver hub [ 0.766125] usbcore: registered new device driver usb [ 0.769262] pps_core: LinuxPPS API ver. 1 registered [ 0.771017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.775086] PTP clock support registered [ 0.777159] EDAC MC: Ver: 3.0.0 [ 0.779057] PCI: Using ACPI for IRQ routing [ 0.781057] NetLabel: Initializing [ 0.782011] NetLabel: domain hash size = 128 [ 0.783010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.784128] NetLabel: unlabeled traffic allowed by default [ 0.786160] vgaarb: loaded [ 0.788355] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.790022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799789] clocksource: Switched to clocksource kvm-clock [ 0.918191] VFS: Disk quotas dquot_6.6.0 [ 0.920476] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.922387] *** VALIDATE ramfs *** [ 0.923516] *** VALIDATE hugetlbfs *** [ 0.925751] pnp: PnP ACPI init [ 0.928192] pnp: PnP ACPI: found 6 devices [ 0.945720] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.948826] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.950735] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.952699] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.954866] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.957058] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.960195] NET: Registered protocol family 2 [ 0.962598] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.967214] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.971404] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.976976] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.981057] TCP: Hash tables configured (established 65536 bind 65536) [ 0.984331] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.987969] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991257] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.994569] NET: Registered protocol family 1 [ 0.997531] RPC: Registered named UNIX socket transport module. [ 0.999908] RPC: Registered udp transport module. [ 1.001844] RPC: Registered tcp transport module. [ 1.003685] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.006513] NET: Registered protocol family 44 [ 1.008684] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.011159] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.013576] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.016352] PCI: CLS 0 bytes, default 64 [ 1.019111] Unpacking initramfs... [ 2.440564] debug: unmapping init [mem 0xffff918e7cc54000-0xffff918e7ffbffff] [ 2.444673] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.446487] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.449408] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.968971] Initialise system trusted keyrings [ 2.971074] Key type blacklist registered [ 2.973111] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.984630] zbud: loaded [ 2.988404] *** VALIDATE nfs *** [ 2.989649] *** VALIDATE nfs4 *** [ 2.991196] pstore: using deflate compression [ 2.995459] Platform Keyring initialized [ 3.104561] NET: Registered protocol family 38 [ 3.106728] Key type asymmetric registered [ 3.108619] Asymmetric key parser 'x509' registered [ 3.110767] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.114249] io scheduler mq-deadline registered [ 3.116140] io scheduler kyber registered [ 3.118307] io scheduler bfq registered [ 3.120613] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.123924] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.126709] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.129582] ACPI: Power Button [PWRF] [ 3.135440] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.142485] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.156594] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.167141] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.185927] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.213744] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.243025] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.247358] Non-volatile memory driver v1.3 [ 3.249128] Linux agpgart interface v0.103 [ 3.277446] virtio_blk virtio1: [vda] 146088 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.280531] vda: detected capacity change from 0 to 74797056 [ 3.295582] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.299249] vdb: detected capacity change from 0 to 1073741824 [ 3.313482] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.316626] vdc: detected capacity change from 0 to 2621440000 [ 3.331456] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.334353] vdd: detected capacity change from 0 to 2621440000 [ 3.350929] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.353509] vde: detected capacity change from 0 to 4294967296 [ 3.369071] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.372244] vdf: detected capacity change from 0 to 4294967296 [ 3.378967] libphy: Fixed MDIO Bus: probed [ 3.384729] usbcore: registered new interface driver usbserial_generic [ 3.386708] usbserial: USB Serial support registered for generic [ 3.388729] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.392335] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.393849] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.395927] mousedev: PS/2 mouse device common for all mice [ 3.399462] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.403479] rtc_cmos 00:05: RTC can wake from S4 [ 3.406441] rtc_cmos 00:05: registered as rtc0 [ 3.406732] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.407984] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.414538] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.415682] intel_pstate: CPU model not supported [ 3.420652] hid: raw HID events driver (C) Jiri Kosina [ 3.423222] usbcore: registered new interface driver usbhid [ 3.425071] usbhid: USB HID core driver [ 3.426480] drop_monitor: Initializing network drop monitor service [ 3.428966] Initializing XFRM netlink socket [ 3.430917] NET: Registered protocol family 10 [ 3.433434] Segment Routing with IPv6 [ 3.434833] NET: Registered protocol family 17 [ 3.437138] mpls_gso: MPLS GSO support [ 3.441880] RAS: Correctable Errors collector initialized. [ 3.444648] AVX version of gcm_enc/dec engaged. [ 3.445740] AES CTR mode by8 optimization enabled [ 3.517837] sched_clock: Marking stable (3517814152, 0)->(4444229884, -926415732) [ 3.527256] registered taskstats version 1 [ 3.529272] Loading compiled-in X.509 certificates [ 3.531246] zswap: loaded using pool lzo/zbud [ 3.561282] Key type big_key registered [ 3.575309] Key type encrypted registered [ 3.577153] ima: No TPM chip found, activating TPM-bypass! [ 3.579589] ima: Allocated hash algorithm: sha1 [ 3.581727] ima: No architecture policies found [ 3.583973] evm: Initialising EVM extended attributes: [ 3.586177] evm: security.selinux [ 3.587720] evm: security.ima [ 3.589089] evm: security.capability [ 3.590709] evm: HMAC attrs: 0x1 [ 3.593377] rtc_cmos 00:05: setting system clock to 2026-08-19 12:18:33 UTC (1787141913) [ 3.600850] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 3.605111] debug: unmapping init [mem 0xffffffffa8b82000-0xffffffffa8e58fff] [ 3.614121] Write protecting the kernel read-only data: 28672k [ 3.618178] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 3.620848] debug: unmapping init [mem 0xffffffffa7b14000-0xffffffffa7bfffff] [ 3.653618] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.660931] systemd[1]: Detected virtualization kvm. [ 3.662872] systemd[1]: Detected architecture x86-64. [ 3.664872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.692355] systemd[1]: No hostname configured. [ 3.694261] systemd[1]: Set hostname to . [ 3.696597] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699428] systemd[1]: Initializing machine ID from random generator. [ 3.739382] random: ln: uninitialized urandom read (6 bytes read) [ 3.829357] random: systemd: uninitialized urandom read (16 bytes read) [ 3.832130] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.840915] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.845908] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ 3.954080] random: fast init done Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.382982] device-mapper: uevent: version 1.0.3 [ 4.386555] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.084191] virtio_net virtio0 ens2: renamed from eth0 [ 5.178245] scsi host0: ata_piix [ 5.188226] scsi host1: ata_piix [ 5.189903] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.192782] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.454610] dracut-initqueue[597]: RTNETLINK answers: File exists [ 10.108651] random: crng init done [ 10.110173] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.500597] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.719967] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.996943] SELinux: Disabled at runtime. [ 12.068685] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.078154] systemd[1]: Detected virtualization kvm. [ 12.080248] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.608893] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.613533] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.619922] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.623662] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.626686] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.633965] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.637832] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ 12.715471] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.102784] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.418665] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.447651] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.641337] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.652966] EDAC sbridge: Ver: 1.1.2 [ 15.330169] Key type dns_resolver registered [ 15.645201] NFS: Registering the id_resolver key type [ 15.647449] Key type id_resolver registered [ 15.649534] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 41.868830] libcfs: loading out-of-tree module taints kernel. [ 41.892738] Key type ._llcrypt registered [ 41.894321] Key type .llcrypt registered [ 41.946224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 54.409634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 56.609316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.635273] alg: No test for adler32 (adler32-zlib) [ 57.282067] hrtimer: interrupt took 8808788 ns [ 58.379229] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 59.635115] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 61.520273] Key type lgssc registered [ 63.927484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.915182] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 126.368044] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 140.819174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.863660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.283684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.352100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.510977] Lustre: lustre-MDT0000: new disk, initializing [ 142.592484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.612466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.605394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.513915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.613977] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.670817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.676317] Lustre: Skipped 1 previous similar message [ 162.817248] Lustre: lustre-MDT0001: new disk, initializing [ 162.950150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.015383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.029986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.010905] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.820707] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.292345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.539483] Lustre: lustre-OST0000: new disk, initializing [ 184.548206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.556889] Lustre: 8438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.650752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.541382] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.056813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.074335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.163880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.650224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.793830] Lustre: lustre-OST0001: new disk, initializing [ 205.798352] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.806328] Lustre: 9511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.886713] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.012542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.029903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.089330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.680981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.502916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.433178] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.463289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.289993] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 253.282332] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 255.646841] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 258.244510] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 259.990612] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 08:22:47 EDT 2026 [ 278.749649] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 280.784857] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.977481] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.978884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.995297] Lustre: Skipped 3 previous similar messages [ 299.009454] Lustre: Skipped 3 previous similar messages [ 303.035879] Lustre: server umount lustre-MDT0000 complete [ 307.483443] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142217 with bad export cookie 4538361211112984138 [ 307.485821] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.498963] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 309.218040] LustreError: 9333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.218486] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.261864] LustreError: 9333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 309.302517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.319799] Lustre: Skipped 1 previous similar message [ 312.229224] LustreError: 6506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.234699] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.275523] Lustre: Skipped 1 previous similar message [ 314.057935] Lustre: server umount lustre-MDT0001 complete [ 318.906850] Lustre: server umount lustre-OST0000 complete [ 322.924713] Lustre: server umount lustre-OST0001 complete [ 329.133874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 338.625889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 377.040694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.397656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.440289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.488234] Lustre: lustre-MDT0000: new disk, initializing [ 377.539643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.549835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.870311] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 392.174610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.307503] Lustre: 15065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 392.379826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 392.385459] Lustre: Skipped 1 previous similar message [ 392.472745] Lustre: lustre-MDT0001: new disk, initializing [ 392.580185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 392.607201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 392.617110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 397.169618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 407.379965] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 413.725276] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 421.338826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 421.687490] Lustre: lustre-OST0000: new disk, initializing [ 421.690795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 421.699420] Lustre: 17034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 421.780353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.186573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 423.198613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 423.313884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 428.056226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 437.682753] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 442.086408] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.275345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 446.727521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 446.907793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 449.002656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.002854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.010136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 449.016460] LustreError: Skipped 1 previous similar message [ 449.031901] Lustre: Skipped 2 previous similar messages [ 454.122027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.130249] Lustre: Skipped 2 previous similar messages [ 463.329359] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 463.489781] Lustre: server umount lustre-OST0000 complete [ 469.476753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.479330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 469.485461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 469.485469] Lustre: Skipped 2 previous similar messages [ 469.502053] Lustre: Skipped 1 previous similar message [ 474.593339] LustreError: 15071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.613505] LustreError: 15071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 475.500476] Lustre: server umount lustre-MDT0000 complete [ 479.172610] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142389 with bad export cookie 4538361211112985671 [ 479.194348] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 479.198661] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 479.529701] Lustre: server umount lustre-MDT0001 complete [ 483.433465] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:26:31 (1787142391) [ 485.967962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 487.755873] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:26:35 (1787142395) [ 494.287367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.891504] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.946594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.854949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 500.198307] LustreError: 19447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.981194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.213449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 519.438714] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 525.335884] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 532.092826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 538.466947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 546.780181] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 559.077969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 559.096262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.115190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.124809] Lustre: Skipped 1 previous similar message [ 561.725140] Lustre: server umount lustre-OST0000 complete [ 569.826085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 569.842475] LustreError: Skipped 1 previous similar message [ 569.851830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.888289] Lustre: Skipped 1 previous similar message [ 573.108561] Lustre: server umount lustre-MDT0000 complete [ 575.461895] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.495388] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 577.326171] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142487 with bad export cookie 4538361211112986546 [ 577.339446] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.361741] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 577.757162] Lustre: server umount lustre-MDT0001 complete [ 591.490586] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 594.879555] Key type lgssc unregistered [ 595.192735] LNet: 22980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.201481] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.223221] LNet: Removed LNI 192.168.203.155@tcp [ 596.310152] Key type .llcrypt unregistered [ 596.327063] Key type ._llcrypt unregistered [ 619.064682] Key type ._llcrypt registered [ 619.067701] Key type .llcrypt registered [ 619.174880] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 620.473326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.503667] alg: No test for adler32 (adler32-zlib) [ 621.530580] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 621.780457] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 623.448199] Key type lgssc registered [ 624.597488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.484349] loop: module loaded [ 633.560275] loop0: detected capacity change from 0 to 204800000 [ 633.768828] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.786359] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.077943] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.061460] loop0: detected capacity change from 0 to 204800000 [ 640.269311] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.283218] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.303677] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.928355] loop0: detected capacity change from 0 to 204800000 [ 649.927751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.954349] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.171785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 651.219463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 651.277442] Lustre: lustre-MDT0000: new disk, initializing [ 651.342609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.365330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 655.403664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 663.521716] loop1: detected capacity change from 0 to 204800000 [ 668.381562] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.767844] Lustre: lustre-OST07c6: new disk, initializing [ 668.772692] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 668.777228] Lustre: Skipped 1 previous similar message [ 668.784381] Lustre: 26163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 668.861484] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 674.687438] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 677.507475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 677.515216] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 677.551251] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 679.097387] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 679.110275] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 679.119870] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 679.129447] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 679.145136] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 679.156248] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 693.673969] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 693.681849] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 693.687962] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.696779] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 693.704182] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 693.709048] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.827442] Lustre: 25150:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 698.836219] Lustre: 25150:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.848620] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.868509] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.883852] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.896769] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.905712] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 698.913046] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.924053] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.933640] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.945177] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.969861] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.254934] Lustre: 26769:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 702.269027] Lustre: 26769:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 702.284292] Lustre: 26769:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 702.298779] Lustre: 26769:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.309794] Lustre: 26769:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.325380] Lustre: 26769:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.334060] Lustre: 26769:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 702.353885] Lustre: 26769:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.362286] Lustre: 26769:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.380272] Lustre: 26769:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.387278] Lustre: 26769:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.409266] Lustre: 26769:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.426038] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.450450] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.463519] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.468769] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.486116] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.504142] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.513753] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.528674] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.542339] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.560671] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.584067] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.601265] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.194454] Lustre: 26769:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 716.203434] Lustre: 26769:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 716.213717] Lustre: 26769:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 716.220214] Lustre: 26769:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.227213] Lustre: 26769:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.236921] Lustre: 26769:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.249209] Lustre: 26769:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 716.255420] Lustre: 26769:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.261049] Lustre: 26769:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.267112] Lustre: 26769:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.273350] Lustre: 26769:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.279180] Lustre: 26769:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 769.512573] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 769.524659] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.547426] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.625265] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.832942] Lustre: server umount lustre-OST07c6 complete [ 785.003516] Lustre: server umount lustre-MDT0000 complete [ 800.748382] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 803.410785] Key type lgssc unregistered [ 803.673090] LNet: 28724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.679462] LNetError: 28724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.708223] LNet: Removed LNI 192.168.203.155@tcp [ 804.476144] Key type .llcrypt unregistered [ 804.481366] Key type ._llcrypt unregistered [ 821.632741] Key type ._llcrypt registered [ 821.635398] Key type .llcrypt registered [ 821.739709] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 822.431094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.449094] alg: No test for adler32 (adler32-zlib) [ 823.604093] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 823.861519] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 825.584514] Key type lgssc registered [ 826.970870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 835.209076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 835.240509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.825350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.602620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 846.322129] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.353988] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 850.564130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.932234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 855.627401] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 863.248408] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.179891] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 876.896085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.275795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.414336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 883.409342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 891.496435] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.334520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.343557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.350928] Lustre: Skipped 1 previous similar message [ 911.352447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.356098] Lustre: Skipped 1 previous similar message [ 916.174485] Lustre: server umount lustre-OST0000 complete [ 921.578292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.593731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.003946] Lustre: server umount lustre-MDT0000 complete [ 926.693058] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.712032] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 929.119912] LustreError: 29981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142839 with bad export cookie 16929872853691755415 [ 929.132311] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.442986] Lustre: server umount lustre-MDT0001 complete [ 939.159325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 941.682450] Key type lgssc unregistered [ 941.915874] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.922348] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.945396] LNet: Removed LNI 192.168.203.155@tcp [ 942.782672] Key type .llcrypt unregistered [ 942.789703] Key type ._llcrypt unregistered [ 960.192958] Key type ._llcrypt registered [ 960.199531] Key type .llcrypt registered [ 960.302335] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 961.369298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.397538] alg: No test for adler32 (adler32-zlib) [ 962.563230] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 962.819737] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 964.496139] Key type lgssc registered [ 965.598315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.028455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.048534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.445433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.806697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 985.065300] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.088674] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 987.378263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.817150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.489626] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 999.594686] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1006.437994] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1013.898529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.408381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.521717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1021.021611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1029.539696] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.473576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1045.476667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.518863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.563629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.566912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.570786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.618783] Lustre: Skipped 1 previous similar message [ 1050.146899] Lustre: server umount lustre-OST0000 complete [ 1053.665702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.678821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.686522] Lustre: Skipped 1 previous similar message [ 1053.698628] Lustre: Skipped 1 previous similar message [ 1058.785150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.804691] Lustre: Skipped 1 previous similar message [ 1059.835533] Lustre: server umount lustre-MDT0000 complete [ 1063.642022] LustreError: 34986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787142973 with bad export cookie 3113873938741950286 [ 1063.660695] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.139535] Lustre: server umount lustre-MDT0001 complete [ 1074.799886] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1077.425792] Key type lgssc unregistered [ 1077.763381] LNet: 38561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.778273] LNetError: 38561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.801703] LNet: Removed LNI 192.168.203.155@tcp [ 1078.749631] Key type .llcrypt unregistered [ 1078.756540] Key type ._llcrypt unregistered [ 1101.699644] Key type ._llcrypt registered [ 1101.702731] Key type .llcrypt registered [ 1101.812622] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1103.031035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1103.041147] alg: No test for adler32 (adler32-zlib) [ 1104.145438] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 1104.388791] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1106.048557] Key type lgssc registered [ 1107.256493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.930376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.996872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.517332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.239402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1127.911378] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.933907] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1133.025486] LustreError: 39981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.577440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.996326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1138.549823] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1145.845142] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1151.591113] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1158.220319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1158.604802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.698407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1164.571224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1171.220906] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1178.086212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.094664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.100196] Lustre: Skipped 2 previous similar messages [ 1178.121238] Lustre: Skipped 1 previous similar message [ 1178.569580] Lustre: server umount lustre-MDT0000 complete [ 1179.312271] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.203538] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.220164] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1183.265400] LustreError: 39959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143093 with bad export cookie 13411008501206212048 [ 1183.271725] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.279193] LustreError: 39959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1184.433667] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 1184.440261] Lustre: Skipped 1 previous similar message [ 1188.322628] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.326649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.352831] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1188.401177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1189.717272] Lustre: server umount lustre-MDT0001 complete [ 1210.530262] Lustre: server umount lustre-OST0000 complete [ 1228.800296] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1231.458559] Key type lgssc unregistered [ 1231.705339] LNet: 43843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.724393] LNetError: 43843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.737868] LNet: Removed LNI 192.168.203.155@tcp [ 1232.517299] Key type .llcrypt unregistered [ 1232.520180] Key type ._llcrypt unregistered [ 1255.373621] Key type ._llcrypt registered [ 1255.378058] Key type .llcrypt registered [ 1255.540101] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1256.451542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1256.604059] alg: No test for adler32 (adler32-zlib) [ 1257.690517] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 1257.934314] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1259.608238] Key type lgssc registered [ 1260.621572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.962776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1268.007539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.584817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.383519] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1279.985317] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.999036] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1282.143659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.535244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.427555] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1293.468274] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.829860] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1305.203364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.519984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.850287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1314.753649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1316.764792] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1322.991073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1323.008495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.020477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1323.695709] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1325.035523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.040829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.060380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1328.576236] Lustre: server umount lustre-OST0000 complete [ 1349.090984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1349.098514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.130517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.135142] Lustre: Skipped 1 previous similar message [ 1352.745220] Lustre: server umount lustre-MDT0000 complete [ 1355.234611] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.267149] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1356.226329] LustreError: 45239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143266 with bad export cookie 7157982315514283859 [ 1356.234839] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1356.238057] LustreError: 45239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1362.646864] Lustre: server umount lustre-MDT0001 complete [ 1373.109184] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1375.887156] Key type lgssc unregistered [ 1376.140485] LNet: 48972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.149790] LNetError: 48972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1376.165793] LNet: Removed LNI 192.168.203.155@tcp [ 1377.125252] Key type .llcrypt unregistered [ 1377.131970] Key type ._llcrypt unregistered [ 1401.536357] Key type ._llcrypt registered [ 1401.539604] Key type .llcrypt registered [ 1401.630750] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1402.452195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1402.467956] alg: No test for adler32 (adler32-zlib) [ 1403.512400] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 1403.713500] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1405.368153] Key type lgssc registered [ 1406.311584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.657512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.679549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.193091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.210665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1425.377980] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.401504] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1428.785635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.179521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1434.048375] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1441.886825] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.343205] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1455.183969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.654263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.775627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1462.340399] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1469.499609] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1477.980404] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1481.248218] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1493.048696] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-19 8h43m20s [ 1494.669368] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1500.648976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.668383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.671574] Lustre: Skipped 2 previous similar messages [ 1500.676157] Lustre: Skipped 1 previous similar message [ 1501.731897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.741362] Lustre: Skipped 3 previous similar messages [ 1502.952909] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1502.964386] Lustre: Skipped 1 previous similar message [ 1507.812152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.828802] Lustre: Skipped 2 previous similar messages [ 1511.392143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1511.564982] Lustre: server umount lustre-MDT0000 complete [ 1513.958758] LustreError: 53081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.984973] LustreError: 53081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1515.155624] LustreError: 53081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.240954] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1520.115880] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.164052] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1522.167081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.328677] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1522.524344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.590677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1523.595917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1527.261568] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1530.766140] Lustre: DEBUG MARKER: Wait for df (36696) ... [ 1532.267399] Lustre: DEBUG MARKER: done [ 1538.018663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1538.043088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.084132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.095734] Lustre: Skipped 1 previous similar message [ 1539.046728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.293612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1549.305772] Lustre: Skipped 5 previous similar messages [ 1551.328136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1551.466192] Lustre: server umount lustre-OST0000 complete [ 1559.525427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1559.526828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.542539] Lustre: Skipped 1 previous similar message [ 1566.698735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.709631] Lustre: Skipped 4 previous similar messages [ 1569.248176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1569.480891] Lustre: server umount lustre-MDT0000 complete [ 1572.837236] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1572.847557] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1573.350923] LustreError: 50373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143483 with bad export cookie 8440627885368948751 [ 1573.357274] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1573.364064] LustreError: 50373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1573.773386] Lustre: server umount lustre-MDT0001 complete [ 1585.612901] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1588.091743] Key type lgssc unregistered [ 1588.412507] LNet: 55324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.418047] LNetError: 55324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.431663] LNet: Removed LNI 192.168.203.155@tcp [ 1589.256801] Key type .llcrypt unregistered [ 1589.263708] Key type ._llcrypt unregistered [ 1629.518743] Key type ._llcrypt registered [ 1629.520698] Key type .llcrypt registered [ 1629.610888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1630.666097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1630.786861] alg: No test for adler32 (adler32-zlib) [ 1631.814962] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 1632.084591] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1633.769578] Key type lgssc registered [ 1635.012914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1642.938421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1642.963313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.363700] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1644.408657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1644.592936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.508595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1658.710494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.979684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1659.068936] Lustre: 57533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1659.381561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1664.319968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1672.332167] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.436613] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1685.527683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.803342] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1686.033989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.547684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1692.863637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1700.455363] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1706.496541] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-19 8h46m53s [ 1708.175612] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1716.022551] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1718.704803] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1718.859627] LustreError: 57539:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1738.943070] Lustre: lustre-MDT0000: Client 5c6ac8c6-7b82-4624-8397-17f404f5b98d (at 192.168.203.55@tcp) reconnecting [ 1758.880377] LustreError: 57539:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1761.282090] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1764.160710] Lustre: DEBUG MARKER: done [ 1774.570024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1774.577287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.592211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1775.081441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.738323] Lustre: server umount lustre-OST0000 complete [ 1782.243297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1782.249876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.264059] Lustre: Skipped 1 previous similar message [ 1782.268627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.283315] Lustre: Skipped 2 previous similar messages [ 1783.331233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.334127] Lustre: Skipped 1 previous similar message [ 1786.402350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.421894] Lustre: Skipped 1 previous similar message [ 1787.945481] Lustre: server umount lustre-MDT0000 complete [ 1792.484439] LustreError: 58552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.538902] LustreError: 58552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1794.178749] LustreError: 58275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143704 with bad export cookie 871173620357896270 [ 1794.179773] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1794.196740] LustreError: 58275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1794.631270] Lustre: server umount lustre-MDT0001 complete [ 1807.796372] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1810.820103] Key type lgssc unregistered [ 1811.160546] LNet: 61865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.177918] LNetError: 61865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.212200] LNet: Removed LNI 192.168.203.155@tcp [ 1812.152579] Key type .llcrypt unregistered [ 1812.155619] Key type ._llcrypt unregistered [ 1857.695168] Key type ._llcrypt registered [ 1857.697154] Key type .llcrypt registered [ 1857.828766] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1859.381124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.431898] alg: No test for adler32 (adler32-zlib) [ 1860.397909] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 1860.644873] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1862.376597] Key type lgssc registered [ 1863.623677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1867.229205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1867.262424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.588480] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1868.608368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1868.749599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.903223] Lustre: Failing over lustre-MDT0000 [ 1870.319206] Lustre: server umount lustre-MDT0000 complete [ 1879.730943] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:49:46 (1787143786) [ 1887.308958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.716596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.684088] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1901.049499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.124737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1901.178481] Lustre: 64664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1901.534092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.411903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1914.648340] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1920.465400] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.019372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.227991] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1927.508807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.866542] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1935.523666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1942.329950] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.670582] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1956.321564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.332827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.350480] Lustre: Skipped 1 previous similar message [ 1956.366178] Lustre: Skipped 2 previous similar messages [ 1961.350344] Lustre: server umount lustre-MDT0000 complete [ 1965.074219] LustreError: 64656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143874 with bad export cookie 9247502076461764570 [ 1965.078997] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.101042] LustreError: 64656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1965.654334] Lustre: server umount lustre-MDT0001 complete [ 1969.167050] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1973.630877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1982.364741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.944265] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143876/real 1787143876] req@00000000826434a2 x1873956065812736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1982.975266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.003019] Lustre: Skipped 1 previous similar message [ 1987.041650] Lustre: 63588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143880/real 1787143880] req@000000006b96f39f x1873956065812864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.128260] Lustre: 63588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143881/real 1787143881] req@00000000faf27729 x1873956065812992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1990.195451] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8055ba24a730d8c4 [ 1990.222483] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 1990.613654] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1990.642340] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1990.749061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.879611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1991.712103] Lustre: 63590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143885/real 1787143885] req@000000002e704a98 x1873956065813248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1992.739478] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.767814] LustreError: 68436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.788475] LustreError: 68436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1995.824455] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1996.768226] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143890/real 1787143890] req@000000003c84e91d x1873956065813376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1998.818163] LustreError: 68437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.838279] LustreError: 68437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2003.943082] LustreError: 68629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.962076] LustreError: 68629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2004.561385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.018455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2009.986957] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2010.098467] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2012.205562] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2017.584420] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2024.286461] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2032.610465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.622118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.629877] Lustre: Skipped 2 previous similar messages [ 2032.640925] Lustre: Skipped 2 previous similar messages [ 2037.731099] LustreError: 68629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.754761] LustreError: 68629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2037.771625] Lustre: server umount lustre-MDT0000 complete [ 2041.924143] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787143951 with bad export cookie 9247502076461766852 [ 2041.926663] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.935226] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.197534] Lustre: server umount lustre-MDT0001 complete [ 2048.888596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2058.208696] Lustre: 63590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787143952/real 1787143952] req@000000001d801033 x1873956065841792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787143968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2058.245676] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.682377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.467165] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8055ba24a730de27 [ 2066.491177] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 2066.820996] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.907744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.972867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2072.042098] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2082.295609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.316576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2087.780884] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2095.038955] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2100.957237] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.507670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.508585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2113.512039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.518957] Lustre: Skipped 1 previous similar message [ 2114.957384] Lustre: server umount lustre-MDT0000 complete [ 2118.501499] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144028 with bad export cookie 9247502076461768231 [ 2118.504869] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.522359] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2118.625327] LustreError: 71122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.626372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2118.665919] LustreError: 71122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2118.680070] Lustre: Skipped 2 previous similar messages [ 2118.897234] Lustre: server umount lustre-MDT0001 complete [ 2125.287978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.906610] Lustre: DEBUG MARKER: files compared the same [ 2131.568623] Lustre: server umount lustre-OST0000 complete [ 2148.702480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2151.199534] Key type lgssc unregistered [ 2151.504940] LNet: 74398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.526586] LNetError: 74398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.542597] LNet: Removed LNI 192.168.203.155@tcp [ 2152.411321] Key type .llcrypt unregistered [ 2152.414354] Key type ._llcrypt unregistered [ 2176.032777] Key type ._llcrypt registered [ 2176.036919] Key type .llcrypt registered [ 2176.202099] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2177.166536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.338157] alg: No test for adler32 (adler32-zlib) [ 2178.499391] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 2178.934143] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2180.672164] Key type lgssc registered [ 2181.710811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.238216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2188.253927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.576549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.601157] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2200.034014] LustreError: 75816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.054127] LustreError: 75816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2205.153443] LustreError: 75815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.792773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.377048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.779733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2220.970580] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.176896] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2234.161976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2234.353743] LustreError: 77637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.475133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.986390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2241.199708] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2249.250353] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2260.459350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2260.475893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.492683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2261.473522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2261.476315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2261.479656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.515434] Lustre: Skipped 1 previous similar message [ 2266.526817] Lustre: server umount lustre-OST0000 complete [ 2271.715255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.728281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.737245] Lustre: Skipped 1 previous similar message [ 2271.756784] Lustre: Skipped 1 previous similar message [ 2276.491227] Lustre: server umount lustre-MDT0000 complete [ 2276.833086] LustreError: 78310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.862059] LustreError: 78310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2280.738180] LustreError: 75795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144190 with bad export cookie 6834208703490359353 [ 2280.738791] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.747077] LustreError: 75795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2281.111905] Lustre: server umount lustre-MDT0001 complete [ 2293.376974] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2296.452878] Key type lgssc unregistered [ 2296.747130] LNet: 79355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.753324] LNetError: 79355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.782266] LNet: Removed LNI 192.168.203.155@tcp [ 2297.597730] Key type .llcrypt unregistered [ 2297.602524] Key type ._llcrypt unregistered [ 2319.944867] Key type ._llcrypt registered [ 2319.947518] Key type .llcrypt registered [ 2320.060858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2321.103754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.148980] alg: No test for adler32 (adler32-zlib) [ 2322.163700] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 2322.467850] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2324.096723] Key type lgssc registered [ 2325.465360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.370421] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2334.395570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2349.856740] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2354.976334] LustreError: 80724:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2381.728305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.896062] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2399.830391] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2410.110379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.260822] Lustre: *** cfs_fail_loc=706, val=0*** [ 2410.627316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.718346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2416.023731] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2416.104817] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.125129] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2417.194799] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.223884] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2422.244851] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.266303] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2426.385095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.882990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.365639] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2440.592874] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2446.727929] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2451.938458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2451.953418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.979953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2452.452702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2452.460172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.572919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.583843] Lustre: Skipped 2 previous similar messages [ 2462.694521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2462.698474] Lustre: Skipped 2 previous similar messages [ 2463.200111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2463.324322] Lustre: server umount lustre-OST0000 complete [ 2467.810374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.811381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.811412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2467.817864] Lustre: Skipped 1 previous similar message [ 2467.839714] Lustre: Skipped 1 previous similar message [ 2472.929144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2472.933046] Lustre: Skipped 1 previous similar message [ 2473.537751] Lustre: server umount lustre-MDT0000 complete [ 2477.577678] LustreError: 80732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144387 with bad export cookie 17206340826513454769 [ 2477.587759] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2477.594499] LustreError: 80732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2478.026982] Lustre: server umount lustre-MDT0001 complete [ 2488.547028] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2491.320511] Key type lgssc unregistered [ 2491.576852] LNet: 84347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.591137] LNetError: 84347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2491.607567] LNet: Removed LNI 192.168.203.155@tcp [ 2492.468746] Key type .llcrypt unregistered [ 2492.472828] Key type ._llcrypt unregistered [ 2516.009614] Key type ._llcrypt registered [ 2516.011657] Key type .llcrypt registered [ 2516.119407] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2516.863401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.896375] alg: No test for adler32 (adler32-zlib) [ 2518.158135] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 2518.522863] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2520.176163] Key type lgssc registered [ 2521.308821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.501200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2528.532817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.152946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2541.882094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.346397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2546.688405] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2547.682897] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.699078] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2552.808181] LustreError: 86444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.833650] LustreError: 86444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2555.067751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.437867] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.472312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.472993] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2560.486589] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.517527] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2561.692315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2565.609014] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.632058] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2569.957753] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.966086] Lustre: MGS: Client bdb697f4-b35c-4aad-b18d-11d6f3be6ca1 (at 0@lo) reconnecting [ 2569.977480] LustreError: 86459:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2570.193962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.286525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2574.422093] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2590.703382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.711168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.719156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.736434] Lustre: Skipped 1 previous similar message [ 2595.043505] Lustre: server umount lustre-OST0000 complete [ 2600.936083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.945227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.947162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.955283] Lustre: Skipped 1 previous similar message [ 2600.966786] Lustre: Skipped 2 previous similar messages [ 2606.050649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.062859] Lustre: Skipped 1 previous similar message [ 2610.148618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2610.158841] Lustre: Skipped 1 previous similar message [ 2613.216139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2613.331979] Lustre: server umount lustre-MDT0000 complete [ 2615.270788] LustreError: 88001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.283536] LustreError: 88001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2617.238186] Lustre: server umount lustre-MDT0001 complete [ 2620.508751] Lustre: server umount MGS complete [ 2630.855988] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:02:18 (1787144538) [ 2656.586983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2665.132374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2701.070636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.434600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2715.632538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.875679] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2715.972238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2716.143336] Lustre: lustre-MDT0001: new disk, initializing [ 2716.269722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.317920] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2716.329281] Lustre: 93644:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2716.348365] LustreError: 93644:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2721.558302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2733.993066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2734.438101] Lustre: lustre-OST0000: new disk, initializing [ 2734.448883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2734.457263] Lustre: 94411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2734.528256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.003537] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2745.954555] Lustre: 94869:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b1fdd103 x1873956755776128/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787144671 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2753.466412] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2761.217948] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2761.236261] Lustre: MGS: Client 38852307-93dd-44c6-aa8a-fe2a8b01b509 (at 0@lo) reconnecting [ 2761.295588] Lustre: 92914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2761.317184] LustreError: 93630:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2761.371328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2761.496300] Lustre: lustre-MDT0000: new disk, initializing [ 2761.676600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.745342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2761.789880] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2762.855943] Lustre: 95481:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000002b390f x1873956755782912/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787144688 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2766.887855] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2767.970926] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2772.025714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2772.047791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2772.058363] Lustre: Skipped 1 previous similar message [ 2772.109484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2798.051426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.056900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.082669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.548495] Lustre: server umount lustre-OST0000 complete [ 2811.878707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2811.880043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.896413] LustreError: Skipped 1 previous similar message [ 2811.910528] Lustre: Skipped 1 previous similar message [ 2811.911522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.911527] Lustre: Skipped 3 previous similar messages [ 2813.629158] Lustre: server umount lustre-MDT0000 complete [ 2816.994131] LustreError: 93616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.031862] LustreError: 93616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2823.644953] Lustre: server umount lustre-MDT0001 complete [ 2827.183511] Lustre: server umount MGS complete [ 2838.893365] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:05:46 (1787144746) [ 2858.470170] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2861.432369] Key type lgssc unregistered [ 2861.737061] LNet: 97982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2861.749321] LNetError: 97982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2861.769856] LNet: Removed LNI 192.168.203.155@tcp [ 2862.595143] Key type .llcrypt unregistered [ 2862.598304] Key type ._llcrypt unregistered [ 2879.660934] Key type ._llcrypt registered [ 2879.662624] Key type .llcrypt registered [ 2879.747574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2880.505070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2880.625735] alg: No test for adler32 (adler32-zlib) [ 2881.622284] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 2881.891905] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2883.632160] Key type lgssc registered [ 2884.854732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.481550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2892.534710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.029387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.437987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2903.523901] LustreError: 99206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.552115] LustreError: 99206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2906.616170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.921581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.261978] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2919.207574] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2925.290245] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2932.467555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.805405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.744892] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2938.930335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2946.233768] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2959.336107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.342919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.371932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.383603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.386568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.394785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.430922] Lustre: Skipped 1 previous similar message [ 2966.500076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.508934] Lustre: Skipped 1 previous similar message [ 2971.104222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2971.265674] Lustre: server umount lustre-OST0000 complete [ 2976.739533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.740691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.750168] Lustre: Skipped 1 previous similar message [ 2976.768463] Lustre: Skipped 1 previous similar message [ 2980.918439] Lustre: server umount lustre-MDT0000 complete [ 2981.858980] LustreError: 99206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.874149] LustreError: 99206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2984.445577] LustreError: 99188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144894 with bad export cookie 6570142561911179170 [ 2984.459882] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.669960] Lustre: server umount lustre-MDT0001 complete [ 2998.178310] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3001.505323] Key type lgssc unregistered [ 3001.976685] LNet: 102744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.984787] LNetError: 102744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3002.001022] LNet: Removed LNI 192.168.203.155@tcp [ 3002.757147] Key type .llcrypt unregistered [ 3002.766872] Key type ._llcrypt unregistered [ 3017.505825] Key type ._llcrypt registered [ 3017.508135] Key type .llcrypt registered [ 3017.596358] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3018.370585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.395786] alg: No test for adler32 (adler32-zlib) [ 3019.480739] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 3019.771770] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3021.472464] Key type lgssc registered [ 3022.700490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3031.224958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3031.232103] LustreError: 103823:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3031.269738] LustreError: 103827:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3031.279982] LustreError: 103823:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 3031.292414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.294425] LustreError: 103823:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3031.294447] LustreError: 103823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3033.028229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.282242] LustreError: 103865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.308626] LustreError: 103865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3043.204542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.637117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.266984] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3051.698886] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 3054.412727] LustreError: 105032:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3054.413723] LustreError: 105033:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3054.431496] LustreError: 105032:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3054.439924] LustreError: 105032:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3054.446116] LustreError: 105032:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3054.451788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.733932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.171621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3059.183399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.214160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.196490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.197418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.221215] Lustre: Skipped 1 previous similar message [ 3063.434075] Lustre: server umount lustre-MDT0000 complete [ 3065.330674] LustreError: 103864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.354424] LustreError: 103864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3066.852338] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787144976 with bad export cookie 10551226676961431684 [ 3066.859585] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3066.860107] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3066.864754] Lustre: 103856:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3067.099343] Lustre: server umount lustre-MDT0001 complete [ 3080.870529] Lustre: server umount lustre-OST0000 complete [ 3088.018750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.377155] LustreError: 106098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.476728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.617945] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3093.479527] LustreError: 106099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.596727] LustreError: 106098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.472077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.933963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3109.708482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3118.606935] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3126.077355] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.613037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.940828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.095223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3139.804595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3147.423787] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3154.414800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.423108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.434840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.438993] Lustre: Skipped 2 previous similar messages [ 3157.476081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3157.477278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3157.491688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.776426] Lustre: server umount lustre-OST0000 complete [ 3164.643972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3164.655621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.670952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.678408] Lustre: Skipped 1 previous similar message [ 3169.934373] Lustre: server umount lustre-MDT0000 complete [ 3172.836596] LustreError: 108677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.263945] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145084 with bad export cookie 10551226676961432370 [ 3174.267427] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.294759] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3174.842486] Lustre: server umount lustre-MDT0001 complete [ 3187.347609] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3190.257195] Key type lgssc unregistered [ 3190.573869] LNet: 109636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3190.586419] LNetError: 109636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3190.605377] LNet: Removed LNI 192.168.203.155@tcp [ 3191.463772] Key type .llcrypt unregistered [ 3191.468429] Key type ._llcrypt unregistered [ 3216.537313] Key type ._llcrypt registered [ 3216.540637] Key type .llcrypt registered [ 3216.682406] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3217.799879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3217.874648] alg: No test for adler32 (adler32-zlib) [ 3218.874727] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 3219.153808] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3220.856716] Key type lgssc registered [ 3222.324924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3229.640370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3229.666556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.167287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.427642] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3241.446441] LustreError: 111056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.474497] LustreError: 111056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3244.168034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.694759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.757058] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3256.277828] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3261.966781] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3268.557138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.866348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.269874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3277.108973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3282.616951] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3287.173227] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3290.711517] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3295.720841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3295.732683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.746437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3296.510854] Lustre: server umount lustre-OST0000 complete [ 3300.834400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3300.846311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.870529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.884387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.888234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.002244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.015134] Lustre: Skipped 2 previous similar messages [ 3310.135958] Lustre: server umount lustre-MDT0000 complete [ 3312.098758] LustreError: 111055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.113768] LustreError: 111055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3314.182038] LustreError: 111035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145224 with bad export cookie 12290982150217667137 [ 3314.183942] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.193258] LustreError: 111035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3314.614377] Lustre: server umount lustre-MDT0001 complete [ 3326.505946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3329.409972] Key type lgssc unregistered [ 3329.723770] LNet: 114695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.736981] LNetError: 114695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3329.756584] LNet: Removed LNI 192.168.203.155@tcp [ 3330.604259] Key type .llcrypt unregistered [ 3330.607857] Key type ._llcrypt unregistered [ 3347.204208] Key type ._llcrypt registered [ 3347.208342] Key type .llcrypt registered [ 3347.314122] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3348.322676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3348.425525] alg: No test for adler32 (adler32-zlib) [ 3349.475465] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 3349.763355] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3351.440502] Key type lgssc registered [ 3352.577372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3358.849723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3358.877391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3360.201398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.313227] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3370.467119] LustreError: 115915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3370.483721] LustreError: 115915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3371.548464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.844175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3375.003000] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3381.659531] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3386.677789] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3393.240857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.605860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.095416] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3405.814723] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3405.887238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3416.039085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.043056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.063812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.072361] Lustre: Skipped 1 previous similar message [ 3416.093124] Lustre: Skipped 1 previous similar message [ 3419.945774] Lustre: server umount lustre-OST0000 complete [ 3426.279163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.291063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.304023] Lustre: Skipped 1 previous similar message [ 3430.471174] Lustre: server umount lustre-MDT0000 complete [ 3431.393298] LustreError: 115912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.408111] LustreError: 115912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3434.609374] LustreError: 115897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145344 with bad export cookie 11380873716073965975 [ 3434.617833] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.627409] LustreError: 115897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3435.040459] Lustre: server umount lustre-MDT0001 complete [ 3444.636424] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3447.249108] Key type lgssc unregistered [ 3447.597723] LNet: 119450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3447.609335] LNetError: 119450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3448.678768] LNet: Removed LNI 192.168.203.155@tcp [ 3449.453499] Key type .llcrypt unregistered [ 3449.458435] Key type ._llcrypt unregistered [ 3474.983813] Key type ._llcrypt registered [ 3474.987484] Key type .llcrypt registered [ 3475.118847] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3476.766379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.793873] alg: No test for adler32 (adler32-zlib) [ 3477.834702] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 3478.065104] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3479.744159] Key type lgssc registered [ 3480.596105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3487.225489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3487.253645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.687460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.811863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3498.741767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.085676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3502.958165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3509.520987] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.211490] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3523.236979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.576421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.045880] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3533.815102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3535.914219] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3543.239784] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3546.482803] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3546.495938] Lustre: Skipped 2 previous similar messages [ 3550.138558] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3550.147744] Lustre: Skipped 2 previous similar messages [ 3552.321487] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3552.346475] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3557.525069] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3557.533569] Lustre: Skipped 2 previous similar messages [ 3560.635572] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 3564.780586] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 3564.787394] Lustre: Skipped 2 previous similar messages [ 3570.240487] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3570.246965] Lustre: Skipped 1 previous similar message [ 3574.756396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3574.757715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.763215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3574.766980] LustreError: Skipped 1 previous similar message [ 3574.780466] Lustre: Skipped 1 previous similar message [ 3579.873941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3579.885276] Lustre: Skipped 2 previous similar messages [ 3580.687867] Lustre: server umount lustre-OST0000 complete [ 3584.995467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.996572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3585.004065] Lustre: Skipped 1 previous similar message [ 3585.016526] Lustre: Skipped 1 previous similar message [ 3589.601225] Lustre: server umount lustre-MDT0000 complete [ 3590.125788] LustreError: 124013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.168585] LustreError: 124013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3593.051883] LustreError: 123500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145502 with bad export cookie 18065981759610246392 [ 3593.053053] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3593.062878] LustreError: 123500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3593.358648] Lustre: server umount lustre-MDT0001 complete [ 3603.869116] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3606.300346] Key type lgssc unregistered [ 3606.569539] LNet: 125708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3606.576797] LNetError: 125708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3606.588082] LNet: Removed LNI 192.168.203.155@tcp [ 3607.193655] Key type .llcrypt unregistered [ 3607.195855] Key type ._llcrypt unregistered [ 3631.189656] Key type ._llcrypt registered [ 3631.196286] Key type .llcrypt registered [ 3631.316451] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3632.353973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3632.475750] alg: No test for adler32 (adler32-zlib) [ 3633.602584] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 3633.801054] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3635.456182] Key type lgssc registered [ 3636.508819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3643.364266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3643.388533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.894477] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3644.931193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.933205] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3648.925627] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3654.125876] LustreError: 127268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.156761] LustreError: 127268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3657.084647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.460824] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3657.483528] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3657.523901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3661.541645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3668.841401] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3674.540095] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.836606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3681.151915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.292978] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3689.339019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3694.118338] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.757596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3702.763247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.768162] LustreError: Skipped 1 previous similar message [ 3702.769529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3702.799419] Lustre: Skipped 1 previous similar message [ 3704.810236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.814326] Lustre: Skipped 2 previous similar messages [ 3708.257655] Lustre: server umount lustre-OST0000 complete [ 3714.016969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3714.025048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.051116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.045619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.160848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.168358] Lustre: Skipped 3 previous similar messages [ 3725.282670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.293125] Lustre: Skipped 1 previous similar message [ 3726.306901] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3726.501800] Lustre: server umount lustre-MDT0000 complete [ 3729.378927] LustreError: 127268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.413302] LustreError: 127268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3729.996640] LustreError: 127248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145639 with bad export cookie 17808079782290197289 [ 3730.000244] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.011886] LustreError: 127248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3730.248304] Lustre: server umount lustre-MDT0001 complete [ 3741.122238] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3744.764377] Key type lgssc unregistered [ 3745.253290] LNet: 130861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3745.267647] LNetError: 130861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3745.293202] LNet: Removed LNI 192.168.203.155@tcp [ 3746.305555] Key type .llcrypt unregistered [ 3746.308395] Key type ._llcrypt unregistered [ 3773.581036] Key type ._llcrypt registered [ 3773.583465] Key type .llcrypt registered [ 3773.706348] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3790.068940] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3791.440910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3791.488185] alg: No test for adler32 (adler32-zlib) [ 3792.510927] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 3792.703381] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3794.368188] Key type lgssc registered [ 3795.350801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3828.029458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3828.073703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.361694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3829.420848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.549232] Lustre: lustre-MDT0000: new disk, initializing [ 3829.683628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.712643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3834.349214] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3844.903200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.081595] Lustre: 134266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3845.140316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3845.144607] Lustre: Skipped 1 previous similar message [ 3845.212276] Lustre: lustre-MDT0001: new disk, initializing [ 3845.337512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.401411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3845.418355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3849.517322] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3858.672497] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3864.843760] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3871.890179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.202384] Lustre: lustre-OST0000: new disk, initializing [ 3872.207110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3872.216238] Lustre: 136236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3872.320365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.808425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3879.504033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3879.516405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3879.648580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3888.795309] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3893.659497] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3893.909937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3898.112360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3898.373211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3900.387756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3900.391139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.404275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3900.409094] Lustre: Skipped 1 previous similar message [ 3905.525559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3905.535499] Lustre: Skipped 2 previous similar messages [ 3906.645512] Lustre: server umount lustre-OST0000 complete [ 3912.161362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3912.167971] LustreError: Skipped 1 previous similar message [ 3912.170226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.190503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.749283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.751334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.776892] Lustre: Skipped 1 previous similar message [ 3917.154303] Lustre: server umount lustre-MDT0000 complete [ 3920.767081] LustreError: 136235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145830 with bad export cookie 11745965579871941153 [ 3920.775561] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.794611] LustreError: 136235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3921.250300] Lustre: server umount lustre-MDT0001 complete [ 3929.678986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.092613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.547180] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3935.201878] LustreError: 138453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.233320] LustreError: 138453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3939.305019] LustreError: 138452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3942.519555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.923484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.084741] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3953.457201] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3958.645375] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3965.046904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.373752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.534852] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3978.407313] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3982.829391] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3983.074551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3988.233535] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3988.440726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3996.176418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3996.294358] Lustre: lustre-OST0001: new disk, initializing [ 3996.299458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3996.309615] Lustre: 141823:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3996.414891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4001.198080] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4003.385986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4003.395325] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4003.490273] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4008.392398] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4008.571990] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4014.195218] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4014.424457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4021.085086] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d9498325800.ost_server_uuid 50 [ 4023.075641] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d9498325800.ost_server_uuid in FULL state after 0 sec [ 4034.022069] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.027908] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4034.028904] Lustre: Skipped 1 previous similar message [ 4034.052460] Lustre: Skipped 1 previous similar message [ 4036.953332] Lustre: server umount lustre-OST0001 complete [ 4039.148691] LustreError: 140274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.179709] LustreError: 140274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4041.188803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4041.211685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.262229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4044.263805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4044.267899] Lustre: Skipped 1 previous similar message [ 4044.277685] LustreError: 140789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.301341] LustreError: 140789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4046.606742] Lustre: server umount lustre-OST0000 complete [ 4054.505258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.520080] Lustre: Skipped 2 previous similar messages [ 4064.743157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.748115] Lustre: Skipped 6 previous similar messages [ 4065.248130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4065.408092] Lustre: server umount lustre-MDT0000 complete [ 4069.681676] LustreError: 138433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787145979 with bad export cookie 11745965579871942028 [ 4069.686251] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.695962] LustreError: 138433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4070.031642] Lustre: server umount lustre-MDT0001 complete [ 4089.057925] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4092.790720] Key type lgssc unregistered [ 4093.219685] LNet: 144829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4093.223634] LNetError: 144829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4094.250856] LNet: Removed LNI 192.168.203.155@tcp [ 4095.135512] Key type .llcrypt unregistered [ 4095.137266] Key type ._llcrypt unregistered [ 4135.495446] Key type ._llcrypt registered [ 4135.497280] Key type .llcrypt registered [ 4135.568925] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4152.570898] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4153.619251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4153.635079] alg: No test for adler32 (adler32-zlib) [ 4154.752903] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 4155.048291] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4156.872862] Key type lgssc registered [ 4158.142917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4192.269931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4192.292966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.596352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4193.630985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4193.709322] Lustre: lustre-MDT0000: new disk, initializing [ 4193.776248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.801097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4198.153437] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4208.605142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.766647] Lustre: 148998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4208.839286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4208.846195] Lustre: Skipped 1 previous similar message [ 4208.960346] Lustre: lustre-MDT0001: new disk, initializing [ 4209.057386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.132095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4209.163443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4213.225285] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4221.867511] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4228.294797] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4236.815141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4237.162831] Lustre: lustre-OST0000: new disk, initializing [ 4237.170486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4237.190690] Lustre: 150966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4237.311784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.052370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4244.063186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4244.209694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4244.322809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4254.539096] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4259.279266] Lustre: Failing over lustre-OST0000 [ 4259.422317] Lustre: server umount lustre-OST0000 complete [ 4275.168716] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787146169/real 1787146169] req@000000008c12a6f2 x1873958471238272/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787146185 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4275.199137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.283663] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787146171/real 1787146171] req@0000000096f63f86 x1873958471239168/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787146187 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4277.316669] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4278.631973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.009622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.026476] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4280.352129] Lustre: 147218:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787146174/real 1787146174] req@00000000b54a6fd0 x1873958471240064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787146190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4285.920367] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787146179/real 1787146179] req@00000000fa0384a4 x1873958471241344/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787146195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4285.960385] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4286.128996] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4286.262744] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4288.759200] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4288.766194] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4290.490356] Lustre: Failing over lustre-MDT0000 [ 4290.734725] Lustre: server umount lustre-MDT0000 complete [ 4291.040919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4291.045180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.051300] Lustre: Skipped 1 previous similar message [ 4291.553386] Lustre: 147219:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787146185/real 1787146185] req@00000000d5d7e43b x1873958471242368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787146201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4291.595828] Lustre: 147219:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4296.678881] LustreError: 149991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.700074] LustreError: 149991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4298.900952] LustreError: 149004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.799057] LustreError: 149005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.822583] LustreError: 149005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4304.026063] LustreError: 149004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.941958] Lustre: 147218:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787146201/real 1787146201] req@00000000a07f1976 x1873958471251584/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787146217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4307.978717] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.154669] LustreError: 152882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.173141] LustreError: 152882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4309.851464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.152845] LustreError: 147216:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b1d99789 x1873958471256832/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4317.175193] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.199044] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4317.441894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.498910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4319.382313] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4322.613527] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4322.800074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4322.809269] Lustre: Skipped 1 previous similar message [ 4322.844369] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4322.881233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4333.027702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.032412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4333.049952] Lustre: Skipped 3 previous similar messages [ 4338.148625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.170334] Lustre: Skipped 2 previous similar messages [ 4343.271469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4343.286378] Lustre: Skipped 1 previous similar message [ 4345.312347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4345.545161] Lustre: server umount lustre-OST0000 complete [ 4353.506927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.508949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4353.514921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4353.526765] Lustre: Skipped 1 previous similar message [ 4358.626635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.633728] Lustre: Skipped 2 previous similar messages [ 4363.232144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4363.455157] Lustre: server umount lustre-MDT0000 complete [ 4363.752554] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.784766] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4367.511444] LustreError: 149791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146277 with bad export cookie 3416106032787771650 [ 4367.519288] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.528730] LustreError: 149791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4367.992539] Lustre: server umount lustre-MDT0001 complete [ 4378.715776] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4381.291355] Key type lgssc unregistered [ 4381.517616] LNet: 154632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4381.523430] LNetError: 154632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4381.551204] LNet: Removed LNI 192.168.203.155@tcp [ 4382.398816] Key type .llcrypt unregistered [ 4382.401441] Key type ._llcrypt unregistered [ 4417.141305] Key type ._llcrypt registered [ 4417.144299] Key type .llcrypt registered [ 4417.241880] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4418.434861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4418.461280] alg: No test for adler32 (adler32-zlib) [ 4419.631411] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 4419.967753] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4421.624178] Key type lgssc registered [ 4422.814348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4430.524430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4430.549809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.035162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.998950] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4441.570388] LustreError: 156152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.594626] LustreError: 156152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4446.700870] LustreError: 156151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.735050] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4450.466914] LustreError: 156180:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4450.475543] LustreError: 156180:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4450.479995] LustreError: 156180:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4450.888161] Lustre: server umount lustre-MDT0000 complete [ 4465.791198] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4475.636981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4514.892550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4515.259038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4515.324846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4515.470794] Lustre: lustre-MDT0000: new disk, initializing [ 4515.614308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.638601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4520.612680] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4531.662498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.800324] Lustre: 159487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4531.878616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4531.882286] Lustre: Skipped 1 previous similar message [ 4531.986253] Lustre: lustre-MDT0001: new disk, initializing [ 4532.200418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4532.263554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4532.276510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4537.301776] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4548.220552] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4554.331823] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4561.650678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.093476] Lustre: lustre-OST0000: new disk, initializing [ 4562.102589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4562.123387] Lustre: 161454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4562.305488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.643123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4568.662301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4568.785548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4569.424430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4578.977677] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4584.025975] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4585.944790] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:34:53 (1787146493) [ 4849.633051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4849.646135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4849.655231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4850.658856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.659755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4850.691659] Lustre: Skipped 2 previous similar messages [ 4855.201828] Lustre: server umount lustre-MDT0000 complete [ 4855.782573] LustreError: 159497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.806021] LustreError: 159497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4859.502613] LustreError: 159478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146769 with bad export cookie 17752950905478636283 [ 4859.505607] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4859.524180] LustreError: 159478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4860.896751] LustreError: 161709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4860.909227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.946866] Lustre: Skipped 1 previous similar message [ 4860.970834] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4864.929667] LustreError: 159494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.929804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4866.018803] Lustre: server umount lustre-MDT0001 complete [ 4876.882992] Lustre: server umount lustre-OST0000 complete [ 4888.358409] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4899.317966] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4938.259393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.593937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4938.647233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4938.774659] Lustre: lustre-MDT0000: new disk, initializing [ 4938.885436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.921856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4943.916897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4954.136526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.275669] Lustre: 166130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4954.347332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4954.357079] Lustre: Skipped 1 previous similar message [ 4954.463757] Lustre: lustre-MDT0001: new disk, initializing [ 4954.617654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4954.655849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4954.667060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4960.007944] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4969.796709] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4975.873326] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4982.691516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4982.980271] Lustre: lustre-OST0000: new disk, initializing [ 4982.983487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4982.997872] Lustre: 168097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4983.124288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.490134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4988.502717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4988.677772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4989.155727] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4998.303214] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5002.960859] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5003.177359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5008.042517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5008.257539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5014.499472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.501587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5014.530258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.547158] Lustre: Skipped 1 previous similar message [ 5016.458467] Lustre: server umount lustre-OST0000 complete [ 5024.738745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.754171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.763168] Lustre: Skipped 2 previous similar messages [ 5024.781689] Lustre: Skipped 1 previous similar message [ 5027.754883] Lustre: server umount lustre-MDT0000 complete [ 5029.872882] LustreError: 167692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.898075] LustreError: 167692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5031.575613] LustreError: 166123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787146941 with bad export cookie 17752950905478795113 [ 5031.578352] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.584872] LustreError: 166123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5031.952864] Lustre: server umount lustre-MDT0001 complete [ 5039.857409] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:42:27 (1787146947) [ 5055.076835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5063.737285] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5101.424847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.717405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5101.758202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5101.885925] Lustre: lustre-MDT0000: new disk, initializing [ 5102.031469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.068627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5106.394649] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5116.631081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.745364] Lustre: 172855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5116.939727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5121.056893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5129.799211] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5135.517406] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5141.923696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5142.248627] Lustre: lustre-OST0000: new disk, initializing [ 5142.256617] Lustre: Skipped 1 previous similar message [ 5142.262824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5142.273794] Lustre: Skipped 2 previous similar messages [ 5142.283735] Lustre: 174834:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5148.587683] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5150.278777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5150.291343] Lustre: Skipped 1 previous similar message [ 5150.299115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5150.409287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5158.645437] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5171.175690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.176639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.179661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.180460] LustreError: Skipped 2 previous similar messages [ 5171.212172] Lustre: Skipped 1 previous similar message [ 5172.509734] Lustre: server umount lustre-OST0000 complete [ 5178.341260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5182.744719] Lustre: server umount lustre-MDT0000 complete [ 5186.536105] LustreError: 172867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.553692] LustreError: 172867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5186.577528] LustreError: 173657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147096 with bad export cookie 17752950905478795995 [ 5186.582703] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5186.590665] LustreError: 173657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5186.865301] Lustre: server umount lustre-MDT0001 complete [ 5195.572444] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:45:03 (1787147103) [ 5210.989994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5219.707856] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5253.391692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.741228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5253.784618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5253.875915] Lustre: lustre-MDT0000: new disk, initializing [ 5253.967113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5253.974037] Lustre: Skipped 2 previous similar messages [ 5254.000242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5258.452901] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5268.344026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5268.553932] Lustre: 179156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5269.009550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5273.924144] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5283.919323] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5290.699851] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5297.830250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5298.053048] Lustre: 181135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5303.289765] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5303.850446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5303.950656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5312.592676] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5319.137863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5319.143136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.148532] Lustre: Skipped 2 previous similar messages [ 5319.153183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5319.157395] Lustre: Skipped 4 previous similar messages [ 5325.037372] Lustre: server umount lustre-OST0000 complete [ 5334.200689] Lustre: server umount lustre-MDT0000 complete [ 5335.009366] LustreError: 180661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.030687] LustreError: 180661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5336.985801] LustreError: 179148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147246 with bad export cookie 17752950905478797073 [ 5336.991369] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5336.997108] LustreError: 179148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5337.281768] Lustre: server umount lustre-MDT0001 complete [ 5346.546707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5348.973416] Key type lgssc unregistered [ 5349.245900] LNet: 182993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5349.265226] LNetError: 182993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5349.280736] LNet: Removed LNI 192.168.203.155@tcp [ 5350.066634] Key type .llcrypt unregistered [ 5350.069671] Key type ._llcrypt unregistered [ 5370.679569] Key type ._llcrypt registered [ 5370.683541] Key type .llcrypt registered [ 5370.782976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5371.641426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5371.672561] alg: No test for adler32 (adler32-zlib) [ 5372.801614] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 5373.011130] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5374.682914] Key type lgssc registered [ 5376.034031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5384.084631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5384.113051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.897410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.337129] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5395.427338] LustreError: 184409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.433906] LustreError: 184409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5400.551112] LustreError: 184410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.034668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.502752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5407.277363] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5414.263446] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5420.231867] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5427.750634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.080801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.145883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5434.194919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5441.131175] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5459.943328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.952127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.970637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5460.963514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5460.973281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5460.979810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.020115] Lustre: Skipped 1 previous similar message [ 5465.856575] Lustre: server umount lustre-OST0000 complete [ 5470.179727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5470.198282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.224675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5475.652479] Lustre: server umount lustre-MDT0000 complete [ 5476.322600] LustreError: 184404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.337190] LustreError: 184404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5479.654218] LustreError: 184390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147389 with bad export cookie 3581803402294908827 [ 5479.660754] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5479.669861] LustreError: 184390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5480.063952] Lustre: server umount lustre-MDT0001 complete [ 5492.088630] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5494.964936] Key type lgssc unregistered [ 5495.312179] LNet: 187967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5495.322791] LNetError: 187967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5495.343051] LNet: Removed LNI 192.168.203.155@tcp [ 5496.183700] Key type .llcrypt unregistered [ 5496.185460] Key type ._llcrypt unregistered [ 5519.288855] Key type ._llcrypt registered [ 5519.294370] Key type .llcrypt registered [ 5519.434664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5520.635276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5520.669927] alg: No test for adler32 (adler32-zlib) [ 5521.675978] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 5521.875186] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5523.512292] Key type lgssc registered [ 5524.625882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5531.831746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5531.848038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.181972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.163415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5543.399072] LustreError: 189385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.428764] LustreError: 189385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5545.386976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.677445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.127985] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5557.821611] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5563.451422] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.548473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.924439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.857799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5579.132902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5582.315045] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5589.473561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.481957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.487433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5589.501721] Lustre: Skipped 1 previous similar message [ 5590.196634] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 5590.205571] Lustre: Skipped 1 previous similar message [ 5594.600063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.357571] Lustre: server umount lustre-OST0000 complete [ 5601.430356] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d94843e1000.ost_server_uuid 50 [ 5602.792148] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5617.638671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5617.647435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.663193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.666552] Lustre: Skipped 1 previous similar message [ 5619.747305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.697501] Lustre: server umount lustre-MDT0000 complete [ 5624.802344] LustreError: 189384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.828174] LustreError: 189384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5626.247373] LustreError: 191964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147536 with bad export cookie 16165204230034569141 [ 5626.260744] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.264539] LustreError: 191964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5626.877647] Lustre: server umount lustre-MDT0001 complete [ 5635.250224] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:52:23 (1787147543) [ 5641.384021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.799989] LustreError: 193301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.911655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.228385] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5647.335159] LustreError: 193302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.432588] LustreError: 193301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.290511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.548603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5657.642523] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5664.111687] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5669.340276] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5676.011694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5676.241094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.314620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5686.510135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5687.892518] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5695.572739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.729592] Lustre: lustre-OST0001: new disk, initializing [ 5695.733691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5695.740830] Lustre: 196141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5695.831847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.810168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5701.828374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5702.069680] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5702.079660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5714.791819] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5719.521359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5719.530935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.579619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5719.592045] Lustre: Skipped 2 previous similar messages [ 5727.726384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.739302] Lustre: Skipped 5 previous similar messages [ 5732.321122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5732.458300] Lustre: server umount lustre-OST0000 complete [ 5732.851624] LustreError: 195124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.869398] LustreError: 195124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5739.830453] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5740.125486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5745.332761] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5745.569634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5751.445648] LustreError: 195125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.481625] LustreError: 195125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5763.571857] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.571995] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.577451] LustreError: Skipped 1 previous similar message [ 5763.581657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5763.586357] Lustre: Skipped 2 previous similar messages [ 5763.607265] Lustre: Skipped 2 previous similar messages [ 5765.627646] Lustre: server umount lustre-OST0001 complete [ 5771.233195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5773.793472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.819442] Lustre: Skipped 1 previous similar message [ 5775.660972] Lustre: server umount lustre-MDT0000 complete [ 5780.460889] LustreError: 193283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147690 with bad export cookie 16165204230034570212 [ 5780.462158] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5780.479834] LustreError: 193283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5781.069593] Lustre: server umount lustre-MDT0001 complete [ 5806.369828] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:55:13 (1787147713) [ 5813.414734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.703105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5813.753221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5813.909919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.832930] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5828.379733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.455399] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5828.480499] Lustre: 199580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5828.527018] LustreError: 199593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.543289] LustreError: 199593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5833.882881] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5842.108653] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5848.612621] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5855.551901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5855.746396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5862.079459] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5864.636421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5869.729702] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5876.184383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5876.266929] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5882.707695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5890.697634] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5895.156126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.169584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.174416] Lustre: Skipped 4 previous similar messages [ 5899.220707] Lustre: server umount lustre-OST0000 complete [ 5900.262265] LustreError: 202904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.272633] LustreError: 202904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5915.105577] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5920.298600] Lustre: server umount lustre-OST0001 complete [ 5929.826361] Lustre: server umount lustre-MDT0000 complete [ 5933.197155] LustreError: 199574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787147843 with bad export cookie 16165204230034571619 [ 5933.201071] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.209220] LustreError: 199574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5933.582450] Lustre: server umount lustre-MDT0001 complete [ 5956.161874] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:57:43 (1787147863) [ 5972.292383] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5981.911935] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6015.208197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.409316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6015.433120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6015.494691] Lustre: lustre-MDT0000: new disk, initializing [ 6015.560470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6015.569755] Lustre: Skipped 3 previous similar messages [ 6015.585386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6020.288888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6031.601037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.734863] Lustre: 207678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6031.868220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6031.872383] Lustre: Skipped 1 previous similar message [ 6032.035102] Lustre: lustre-MDT0001: new disk, initializing [ 6032.183745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6032.194071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6036.939319] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6048.990364] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.198087] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6063.415012] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.756044] Lustre: lustre-OST0000: new disk, initializing [ 6063.762277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6063.770687] Lustre: 209650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6071.845800] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6072.426767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6072.432804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6072.489924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6082.479503] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6088.416778] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6088.684459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6093.486337] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6093.735713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6098.411753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.423384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6098.424663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6098.424669] Lustre: Skipped 5 previous similar messages [ 6098.430270] Lustre: Skipped 6 previous similar messages [ 6098.459304] LustreError: Skipped 3 previous similar messages [ 6110.176139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6110.374976] Lustre: server umount lustre-OST0000 complete [ 6120.325225] Lustre: server umount lustre-MDT0000 complete [ 6123.747037] LustreError: 207670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148033 with bad export cookie 16165204230034573180 [ 6123.750909] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6123.756205] LustreError: 207670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6124.101335] Lustre: server umount lustre-MDT0001 complete [ 6131.881373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.249719] LustreError: 211867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.269449] LustreError: 211867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6136.546747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6144.124195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.464415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6144.475355] Lustre: Skipped 3 previous similar messages [ 6148.132446] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6154.669657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6159.171198] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6164.720829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.100751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6178.243747] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6182.186344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6182.351246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6186.939978] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6187.250396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6190.050033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6190.083914] LustreError: Skipped 1 previous similar message [ 6190.093518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.129426] Lustre: Skipped 2 previous similar messages [ 6195.676288] Lustre: server umount lustre-OST0000 complete [ 6203.469125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6208.198742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6237.150831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.407883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6243.564385] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.584818] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6250.996330] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.852160] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6259.169213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6259.181863] LustreError: Skipped 1 previous similar message [ 6259.188451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.201064] Lustre: Skipped 11 previous similar messages [ 6261.114740] Lustre: server umount lustre-OST0000 complete [ 6270.972430] Lustre: server umount lustre-MDT0000 complete [ 6274.296459] LustreError: 211847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148184 with bad export cookie 16165204230034574055 [ 6274.308452] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.316737] LustreError: 211847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6274.615331] Lustre: server umount lustre-MDT0001 complete [ 6282.738222] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:03:10 (1787148190) [ 6289.115772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.663452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6301.635129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.188312] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6315.485485] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.522670] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6328.465417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.171472] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6345.161105] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6350.977405] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6351.169757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6355.787185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6356.019891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6363.155901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.471872] Lustre: lustre-OST0001: new disk, initializing [ 6363.477483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6363.490285] Lustre: 221216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6369.874774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6371.900153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6371.912745] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6372.055072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6379.692951] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6384.312103] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6384.515815] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6388.898642] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6389.069282] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6392.808619] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6392.809249] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.817423] LustreError: Skipped 1 previous similar message [ 6392.827602] Lustre: Skipped 4 previous similar messages [ 6397.228685] Lustre: server umount lustre-OST0001 complete [ 6397.936225] LustreError: 219666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6397.967514] LustreError: 219666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6404.187407] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6404.382246] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6409.130616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6409.423502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6437.503750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.726347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6437.733110] Lustre: Skipped 6 previous similar messages [ 6442.989833] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6443.003626] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6443.004544] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6443.017830] Lustre: Skipped 1 previous similar message [ 6443.387056] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6450.790757] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.333842] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6460.061704] Lustre: server umount lustre-OST0001 complete [ 6482.541425] Lustre: server umount lustre-OST0000 complete [ 6494.687826] Lustre: server umount lustre-MDT0000 complete [ 6498.467090] LustreError: 218581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148408 with bad export cookie 16165204230034575021 [ 6498.472028] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.474991] LustreError: 218581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6504.952586] Lustre: server umount lustre-MDT0001 complete [ 6529.340797] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:07:16 (1787148436) [ 6537.183131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.518528] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6537.554719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6542.811398] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6551.159718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.250907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6551.273609] Lustre: 226456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6556.397519] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6564.241640] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6571.074535] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.709877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.879679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6582.941515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6589.533956] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6597.727162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6597.833885] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6603.403542] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6613.225945] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.281510] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6622.140036] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6622.460427] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6628.304775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6628.633388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6634.588669] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d94897cf800.ost_server_uuid 50 [ 6636.162820] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d94897cf800.ost_server_uuid in IDLE state after 0 sec [ 6637.141938] Lustre: Permanently deactivating lustre-OST0001 [ 6637.160841] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6640.140571] Lustre: Permanently reactivating lustre-OST0001 [ 6647.786193] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6647.789059] Lustre: Skipped 17 previous similar messages [ 6649.916988] Lustre: server umount lustre-OST0001 complete [ 6652.897369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.901455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.902069] LustreError: Skipped 4 previous similar messages [ 6652.916979] Lustre: Skipped 10 previous similar messages [ 6667.233113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6667.353294] Lustre: server umount lustre-OST0000 complete [ 6679.887778] Lustre: server umount lustre-MDT0000 complete [ 6684.667851] LustreError: 226451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148594 with bad export cookie 16165204230034576309 [ 6684.683982] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6684.693045] LustreError: 226451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6685.049376] Lustre: server umount lustre-MDT0001 complete [ 6708.934689] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:10:16 (1787148616) [ 6716.915292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.259333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6717.300201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6717.314283] Lustre: Skipped 5 previous similar messages [ 6721.389839] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6729.855581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.934087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6729.962430] Lustre: 233148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6734.546401] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6743.092314] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6749.603722] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6757.205576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6757.372083] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6757.387980] Lustre: Permanently deactivating lustre-OST0000 [ 6757.396157] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6764.609622] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6766.647042] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6773.822145] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.494683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6782.670672] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6788.225728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6789.243394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6797.648872] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6799.741420] Lustre: Permanently reactivating lustre-OST0000 [ 6799.756236] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6799.766559] Lustre: Skipped 2 previous similar messages [ 6808.236708] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6808.255158] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 0000000021969e7f, cur 1787148718 deadline 1787148706 last 1787148676 [ 6808.310869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6817.292923] Lustre: server umount lustre-OST0001 complete [ 6827.388611] Lustre: server umount lustre-OST0000 complete [ 6845.408253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6845.696471] Lustre: server umount lustre-MDT0000 complete [ 6849.832894] LustreError: 233892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148759 with bad export cookie 16165204230034577961 [ 6849.842021] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.847783] LustreError: 233892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6850.444889] Lustre: server umount lustre-MDT0001 complete [ 6862.676377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6865.774398] Key type lgssc unregistered [ 6866.099421] LNet: 238033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6866.115462] LNetError: 238033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6866.139715] LNet: Removed LNI 192.168.203.155@tcp [ 6866.976239] Key type .llcrypt unregistered [ 6866.977931] Key type ._llcrypt unregistered [ 6890.734812] Key type ._llcrypt registered [ 6890.737726] Key type .llcrypt registered [ 6890.905516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6892.432704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6892.485907] alg: No test for adler32 (adler32-zlib) [ 6893.550349] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 6893.813306] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6895.552164] Key type lgssc registered [ 6896.629699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6906.992086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6907.015538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.673853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.210767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6918.119956] LustreError: 239499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.134436] LustreError: 239499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6922.234611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.378232] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6922.403290] Lustre: 239490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6922.412959] Lustre: Permanently deactivating lustre-MDT0001 [ 6922.423822] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6922.712618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6926.040475] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6926.056473] LustreError: 239531:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6926.068214] LustreError: 239531:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6926.085543] LustreError: 239531:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6927.815338] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6936.182864] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6942.397826] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6950.702548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6951.021101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.094316] LustreError: 241341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.107371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6957.877354] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6959.628968] LustreError: 241342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.707821] LustreError: 241341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.107815] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6967.822846] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6972.902473] LustreError: 241340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.926593] LustreError: 241340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6974.553862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6974.903244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6980.104698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6980.126164] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6984.044502] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6993.803248] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6997.027500] Lustre: Permanently reactivating lustre-MDT0001 [ 6997.076492] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6997.078896] Lustre: Skipped 2 previous similar messages [ 7001.815376] Lustre: lustre-MDT0001: Client 4fa7f298-a21f-4c0b-bb78-7f3d3980049a (at 192.168.203.55@tcp) reconnecting [ 7007.535451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7009.242531] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7014.841290] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7015.080311] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7019.788266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7020.026815] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7022.612212] Lustre: Permanently deactivating lustre-MDT0001 [ 7022.646876] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7022.665189] Lustre: Skipped 2 previous similar messages [ 7032.462614] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7040.480885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7040.503137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.543330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.506419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.508190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.534228] Lustre: Skipped 1 previous similar message [ 7041.566070] Lustre: Skipped 3 previous similar messages [ 7046.013454] Lustre: server umount lustre-MDT0000 complete [ 7046.637468] LustreError: 239499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.655316] LustreError: 239499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7049.883689] LustreError: 239481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787148959 with bad export cookie 4810715569107515484 [ 7049.894635] LustreError: 239481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7049.896538] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.288272] Lustre: server umount lustre-MDT0001 complete [ 7056.150075] Lustre: server umount lustre-OST0000 complete [ 7067.533772] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:16:15 (1787148975) [ 7078.854669] Lustre: server umount lustre-OST0001 complete [ 7088.499479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7101.761742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7139.139393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.535980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7139.553418] Lustre: Skipped 2 previous similar messages [ 7139.577425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7139.676745] Lustre: lustre-MDT0000: new disk, initializing [ 7139.743303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.763787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7144.679657] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7157.711319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.895032] Lustre: 247781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7157.994782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7157.998658] Lustre: Skipped 1 previous similar message [ 7158.101293] Lustre: lustre-MDT0001: new disk, initializing [ 7158.274314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7158.317019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7158.364358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7163.901688] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7172.884995] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.272431] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7185.243264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.536344] Lustre: lustre-OST0000: new disk, initializing [ 7185.543224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7185.555366] Lustre: 249753:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7185.688513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.068937] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7195.202599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7195.213424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7195.290836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7203.137655] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7205.157850] LustreError: 247789:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7214.519251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7214.752974] Lustre: lustre-OST0001: new disk, initializing [ 7214.765643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7214.779598] Lustre: 250868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7215.169199] LustreError: 247789:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7215.200328] LustreError: 248798:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7220.855976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7220.875621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7220.965731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7221.381713] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7225.232466] LustreError: 248798:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7232.146493] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7234.041542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7236.581860] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7236.591697] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.603366] Lustre: Skipped 2 previous similar messages [ 7236.604065] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7241.703169] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7241.710522] Lustre: Skipped 2 previous similar messages [ 7242.358244] Lustre: server umount lustre-OST0001 complete [ 7246.818095] LustreError: 249737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.847922] LustreError: 249737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7262.177697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7262.185363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.193586] LustreError: Skipped 1 previous similar message [ 7262.197752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7262.224709] Lustre: Skipped 1 previous similar message [ 7266.522497] Lustre: server umount lustre-OST0000 complete [ 7270.884243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.895404] LustreError: Skipped 1 previous similar message [ 7270.901642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.911879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.920729] Lustre: Skipped 1 previous similar message [ 7276.613150] Lustre: server umount lustre-MDT0000 complete [ 7280.357228] LustreError: 247772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787149190 with bad export cookie 4810715569107518298 [ 7280.359532] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7280.371083] LustreError: 247772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7286.761116] Lustre: server umount lustre-MDT0001 complete [ 7298.303323] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7301.429062] Key type lgssc unregistered [ 7301.804719] LNet: 252907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7301.834554] LNetError: 252907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7301.852967] LNet: Removed LNI 192.168.203.155@tcp [ 7302.822767] Key type .llcrypt unregistered [ 7302.831770] Key type ._llcrypt unregistered [ 7345.323893] Key type ._llcrypt registered [ 7345.326828] Key type .llcrypt registered [ 7345.438550] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7346.794652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7346.838177] alg: No test for adler32 (adler32-zlib) [ 7347.843305] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 7348.056795] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7349.712153] Key type lgssc registered [ 7350.741951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7358.517353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7358.527368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.814224] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7359.842629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7359.953097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.734329] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7373.080143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.201427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7373.239818] Lustre: 255113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7373.463290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7377.688337] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7384.551341] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7390.174695] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7397.084598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.288492] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7397.462869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.706576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7407.271975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7411.459378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7463.906971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7463.914414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.955050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7468.514455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7468.531528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.552759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7469.963832] Lustre: server umount lustre-OST0000 complete [ 7474.197579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7487.375540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.605513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.897270] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7494.781077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7494.791148] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7494.791321] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7501.368318] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.222800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7541.225670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.232468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.239393] Lustre: Skipped 1 previous similar message [ 7541.248345] Lustre: Skipped 2 previous similar messages [ 7546.299601] Lustre: server umount lustre-OST0000 complete [ 7551.459875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.487937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7551.498627] Lustre: Skipped 2 previous similar messages [ 7556.578561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.585953] Lustre: Skipped 1 previous similar message [ 7564.256193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7564.624491] Lustre: server umount lustre-MDT0000 complete [ 7568.999265] LustreError: 255108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787149478 with bad export cookie 4935471915627672748 [ 7569.012800] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.296722] Lustre: server umount lustre-MDT0001 complete [ 7580.820808] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7583.635644] Key type lgssc unregistered [ 7583.891311] LNet: 260548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7583.899281] LNetError: 260548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7583.925456] LNet: Removed LNI 192.168.203.155@tcp [ 7584.665165] Key type .llcrypt unregistered [ 7584.667408] Key type ._llcrypt unregistered [ 7608.664874] Key type ._llcrypt registered [ 7608.668540] Key type .llcrypt registered [ 7608.784670] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7610.002101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7610.053872] alg: No test for adler32 (adler32-zlib) [ 7611.182728] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 7611.411215] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7613.096214] Key type lgssc registered [ 7614.371500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7621.073994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7621.088488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.606141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7627.432494] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7632.866175] LustreError: 261966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7632.890056] LustreError: 261966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7636.282473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.735774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7641.513388] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7650.626371] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7657.399967] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7664.889396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7665.326807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7667.379803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7672.152172] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7680.074753] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7707.625345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7707.629335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.646437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7707.655656] Lustre: Skipped 1 previous similar message [ 7711.925549] Lustre: server umount lustre-OST0000 complete [ 7717.860630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.900068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7717.908151] Lustre: Skipped 2 previous similar messages [ 7722.522727] Lustre: server umount lustre-MDT0000 complete [ 7722.983930] LustreError: 261962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7723.013450] LustreError: 261962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7726.572462] LustreError: 261947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787149636 with bad export cookie 15589125862693305041 [ 7726.586740] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7727.013782] Lustre: server umount lustre-MDT0001 complete [ 7738.990103] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7742.055607] Key type lgssc unregistered [ 7742.330700] LNet: 266276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7742.347283] LNetError: 266276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7742.363359] LNet: Removed LNI 192.168.203.155@tcp [ 7743.257365] Key type .llcrypt unregistered [ 7743.259917] Key type ._llcrypt unregistered [ 7762.833893] Key type ._llcrypt registered [ 7762.840562] Key type .llcrypt registered [ 7762.956404] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7764.072534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7764.230312] alg: No test for adler32 (adler32-zlib) [ 7765.359637] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 7765.645273] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7767.408148] Key type lgssc registered [ 7768.839380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7777.847828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7777.864658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.384492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7784.380577] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7789.540940] LustreError: 267497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.555679] LustreError: 267497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7792.406814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.674156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7796.337719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7802.958197] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7809.029650] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7815.772341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.123280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.810764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7824.364392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7829.434178] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7842.785491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7842.795060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.824285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.848521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.856611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7844.858557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.863653] Lustre: Skipped 1 previous similar message [ 7848.258682] Lustre: server umount lustre-OST0000 complete [ 7854.048984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7854.054892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.070776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.446377] Lustre: server umount lustre-MDT0000 complete [ 7860.194170] LustreError: 267498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7860.224546] LustreError: 267498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7862.373963] LustreError: 267478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787149772 with bad export cookie 16147298576684191137 [ 7862.376940] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.389974] LustreError: 267478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7862.733261] Lustre: server umount lustre-MDT0001 complete [ 7873.838562] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7876.538993] Key type lgssc unregistered [ 7876.796754] LNet: 271177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7876.803321] LNetError: 271177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7876.842176] LNet: Removed LNI 192.168.203.155@tcp [ 7877.809073] Key type .llcrypt unregistered [ 7877.812576] Key type ._llcrypt unregistered [ 7906.219504] Key type ._llcrypt registered [ 7906.224777] Key type .llcrypt registered [ 7906.306961] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7907.434527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7907.453775] alg: No test for adler32 (adler32-zlib) [ 7908.527258] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 7908.780198] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7910.497263] Key type lgssc registered [ 7911.732705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7919.765393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7919.786215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.143290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7925.544804] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7931.371175] LustreError: 272598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.393341] LustreError: 272598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7934.116829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.613430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7938.986933] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7946.555973] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7952.651657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7959.501351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7959.939941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.753496] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7967.093779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7972.648438] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8003.041434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.053469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.088976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8005.093165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8005.094404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8005.127946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.842236] Lustre: server umount lustre-OST0000 complete [ 8011.238869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8011.249875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8011.269858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8011.272719] Lustre: Skipped 1 previous similar message [ 8015.329806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.333049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.359525] Lustre: Skipped 1 previous similar message [ 8017.556495] Lustre: server umount lustre-MDT0000 complete [ 8020.457043] LustreError: 272597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.487301] LustreError: 272597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8021.690902] LustreError: 272578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787149931 with bad export cookie 13788056642393147718 [ 8021.693662] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8021.710790] LustreError: 272578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8022.055313] Lustre: server umount lustre-MDT0001 complete [ 8032.188367] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8034.742196] Key type lgssc unregistered [ 8035.042854] LNet: 276909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8035.053202] LNetError: 276909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8035.079759] LNet: Removed LNI 192.168.203.155@tcp [ 8035.853733] Key type .llcrypt unregistered [ 8035.857237] Key type ._llcrypt unregistered [ 8053.008538] Key type ._llcrypt registered [ 8053.010321] Key type .llcrypt registered [ 8053.106990] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8054.088819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8054.101456] alg: No test for adler32 (adler32-zlib) [ 8055.235520] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 8055.533130] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8057.176168] Key type lgssc registered [ 8058.252643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.512856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8065.537326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.102728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.297802] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8076.261809] LustreError: 278142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.273728] LustreError: 278142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8081.382143] LustreError: 278144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.487488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.862898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8086.679894] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8094.130920] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8100.029884] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8107.910865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8108.365781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.479271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8114.734929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8122.516128] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8138.210514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.228473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.247243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8143.331887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8143.336743] Lustre: Skipped 1 previous similar message [ 8144.508723] Lustre: server umount lustre-OST0000 complete [ 8149.474476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8149.491882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.509732] Lustre: Skipped 1 previous similar message [ 8149.519777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.525278] Lustre: Skipped 1 previous similar message [ 8153.576059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.581580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8153.598401] Lustre: Skipped 1 previous similar message [ 8154.897581] Lustre: server umount lustre-MDT0000 complete [ 8158.644429] LustreError: 278108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150068 with bad export cookie 7684715018951555508 [ 8158.655213] LustreError: 278108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8158.657929] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.156804] Lustre: server umount lustre-MDT0001 complete [ 8168.892218] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8171.168748] Key type lgssc unregistered [ 8171.430635] LNet: 281847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8171.437405] LNetError: 281847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8171.454826] LNet: Removed LNI 192.168.203.155@tcp [ 8172.208388] Key type .llcrypt unregistered [ 8172.210720] Key type ._llcrypt unregistered [ 8182.215770] Key type ._llcrypt registered [ 8182.221199] Key type .llcrypt registered [ 8182.328794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8198.618073] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8213.407762] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8214.078924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8214.135470] alg: No test for adler32 (adler32-zlib) [ 8215.126202] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 8215.361491] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8217.016226] Key type lgssc registered [ 8217.921690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8250.637565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8250.677803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.892616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8251.916570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8251.991957] Lustre: lustre-MDT0000: new disk, initializing [ 8252.078672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8252.107861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8256.177984] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8266.418568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.589293] Lustre: 285402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8266.687130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8266.697742] Lustre: Skipped 1 previous similar message [ 8266.785562] Lustre: lustre-MDT0001: new disk, initializing [ 8266.872520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.940158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8266.947815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8270.864104] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8279.890661] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8285.391550] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.474295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.696374] Lustre: lustre-OST0000: new disk, initializing [ 8291.700169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8291.705238] Lustre: 287368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8291.772526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.374519] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8297.565835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8297.579070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8297.750713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8305.783691] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8310.177675] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8310.417712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8314.859952] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8315.143471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8318.442838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8318.443031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.456591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.461550] LustreError: Skipped 1 previous similar message [ 8318.494170] Lustre: Skipped 1 previous similar message [ 8323.558436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.565553] Lustre: Skipped 1 previous similar message [ 8328.676832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.686354] Lustre: Skipped 2 previous similar messages [ 8331.233421] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8331.417608] Lustre: server umount lustre-OST0000 complete [ 8338.402194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8338.412300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8338.429150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8341.924604] Lustre: server umount lustre-MDT0000 complete [ 8346.058418] LustreError: 285393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150255 with bad export cookie 15779013059762824787 [ 8346.068588] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.074469] LustreError: 285393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8346.475205] Lustre: server umount lustre-MDT0001 complete [ 8355.673566] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:37:43 (1787150263) [ 8362.575573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.165311] LustreError: 289780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.191963] LustreError: 289780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8363.297792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.369698] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8368.614956] LustreError: 289781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.730634] LustreError: 289780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.558702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.913249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8381.295747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8389.949541] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8396.168859] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8402.659964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.089414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.072415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8416.002621] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8430.059029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8430.071207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8430.097725] Lustre: Skipped 1 previous similar message [ 8430.113260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.125772] Lustre: Skipped 1 previous similar message [ 8432.100691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.104796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.341848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.351157] Lustre: Skipped 4 previous similar messages [ 8443.360413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8443.534944] Lustre: server umount lustre-OST0000 complete [ 8447.458314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.472661] Lustre: Skipped 1 previous similar message [ 8453.509871] Lustre: server umount lustre-MDT0000 complete [ 8456.980380] LustreError: 289760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150366 with bad export cookie 15779013059762825669 [ 8456.997990] LustreError: 289760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8457.002732] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.406509] Lustre: server umount lustre-MDT0001 complete [ 8467.259491] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8470.125677] Key type lgssc unregistered [ 8470.458925] LNet: 293344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8470.471265] LNetError: 293344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8470.500347] LNet: Removed LNI 192.168.203.155@tcp [ 8471.380262] Key type .llcrypt unregistered [ 8471.387866] Key type ._llcrypt unregistered [ 8495.630107] Key type ._llcrypt registered [ 8495.636845] Key type .llcrypt registered [ 8495.721961] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8508.193753] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8509.269173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8509.323102] alg: No test for adler32 (adler32-zlib) [ 8510.456241] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 8510.757624] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8512.514368] Key type lgssc registered [ 8513.618352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8557.119677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8557.149886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.356549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8558.385574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8558.457648] Lustre: lustre-MDT0000: new disk, initializing [ 8558.580844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8558.601413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8562.889262] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8573.201357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.321307] Lustre: 297153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8573.406632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8573.414692] Lustre: Skipped 1 previous similar message [ 8573.494209] Lustre: lustre-MDT0001: new disk, initializing [ 8573.612257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8573.654207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8573.666828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8577.930879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8587.459692] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8593.113401] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8599.684370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8600.040412] Lustre: lustre-OST2710: new disk, initializing [ 8600.045478] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8600.054980] Lustre: 299119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8600.176465] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8606.342863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8607.321806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8607.333435] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8607.459501] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8615.757198] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8622.302741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.470602] Lustre: lustre-OST03e8: new disk, initializing [ 8622.475442] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8622.485183] Lustre: 300185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8622.596663] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8628.340851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8628.359203] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8628.402754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8628.449665] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8635.572394] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8641.276747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8641.491761] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8646.023278] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8646.202593] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8651.157963] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8651.408930] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8656.194334] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8656.433584] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8660.961721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8660.968142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.994043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8662.340798] Lustre: server umount lustre-MDT0000 complete [ 8664.546540] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.569298] Lustre: Skipped 2 previous similar messages [ 8665.144728] LustreError: 297143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150575 with bad export cookie 10044568452587653086 [ 8665.149932] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8665.154444] LustreError: 297143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8665.444620] Lustre: server umount lustre-MDT0001 complete [ 8679.042583] Lustre: server umount lustre-OST2710 complete [ 8693.267550] Lustre: server umount lustre-OST03e8 complete [ 8711.513145] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8722.480918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8764.268811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.576567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8764.623733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8764.713350] Lustre: lustre-MDT0000: new disk, initializing [ 8764.803819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.829954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8770.205410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8780.228308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.364180] Lustre: 305435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8780.442289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8780.454315] Lustre: Skipped 1 previous similar message [ 8780.569745] Lustre: lustre-MDT0001: new disk, initializing [ 8780.693443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.727221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8780.733861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8784.539206] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8794.524933] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8800.250641] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8807.435664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.777132] Lustre: lustre-OST0000: new disk, initializing [ 8807.780531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8807.793544] Lustre: 307404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8807.880763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.212060] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8817.725557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8817.735780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8817.837923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8822.867228] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8827.211645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8827.457605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8831.519515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8831.777971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8838.633095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8838.650135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.680519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.693770] Lustre: Skipped 1 previous similar message [ 8840.510765] Lustre: server umount lustre-OST0000 complete [ 8847.334298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8847.343570] LustreError: Skipped 1 previous similar message [ 8847.351435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.361427] Lustre: Skipped 1 previous similar message [ 8847.367823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8851.426406] Lustre: server umount lustre-MDT0000 complete [ 8853.985709] LustreError: 306248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.018285] LustreError: 306248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8854.935834] LustreError: 305427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150764 with bad export cookie 10044568452587654766 [ 8854.936340] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8854.957463] LustreError: 305427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8855.225327] Lustre: server umount lustre-MDT0001 complete [ 8865.260830] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:46:12 (1787150772) [ 8867.031138] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8868.655775] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:46:16 (1787150776) [ 8871.554375] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8898.765573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.008432] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8899.029405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8899.158480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.442700] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8911.734589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.821447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8911.859942] Lustre: 311262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8916.701855] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8937.789451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8938.070641] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 8938.075835] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8938.082431] LustreError: 312717:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8938.089395] LustreError: 312717:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8938.092793] LustreError: 312717:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8938.153429] Lustre: server umount lustre-OST0000 complete [ 8938.163907] LustreError: 312717:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8947.175599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.185171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8947.191903] Lustre: Skipped 2 previous similar messages [ 8947.202453] Lustre: Skipped 3 previous similar messages [ 8951.099641] Lustre: server umount lustre-MDT0000 complete [ 8952.291603] LustreError: 311274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8952.313881] LustreError: 311274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8954.380803] LustreError: 311256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787150864 with bad export cookie 10044568452587655648 [ 8954.394394] LustreError: 311256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8954.394433] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8954.727390] Lustre: server umount lustre-MDT0001 complete [ 8966.115451] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8968.484436] Key type lgssc unregistered [ 8968.712832] LNet: 313842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8968.724411] LNetError: 313842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8968.743184] LNet: Removed LNI 192.168.203.155@tcp [ 8969.405357] Key type .llcrypt unregistered [ 8969.407398] Key type ._llcrypt unregistered [ 8981.475939] Key type ._llcrypt registered [ 8981.479915] Key type .llcrypt registered [ 8981.564201] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8982.231656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8982.244452] alg: No test for adler32 (adler32-zlib) [ 8983.302383] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 8983.614772] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8985.328158] Key type lgssc registered [ 8986.385854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9034.803557] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9046.193626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9046.203368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.398793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9047.473633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9047.655842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9052.264742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9060.792110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9060.904067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9060.944899] Lustre: 316609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9061.332209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9065.958959] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9073.263718] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9078.798887] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9098.120072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9098.355968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9098.526622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9103.877047] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9110.567689] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9115.618694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9115.625122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.650155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9116.144062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9116.159035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9116.179223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9116.193255] Lustre: Skipped 1 previous similar message [ 9118.706744] Lustre: server umount lustre-OST0000 complete [ 9122.789204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9122.798347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.825701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9126.370468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9126.371334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9130.125853] Lustre: server umount lustre-MDT0000 complete [ 9133.583623] LustreError: 318471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151043 with bad export cookie 17918509323989884533 [ 9133.592172] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.601900] LustreError: 318471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9134.014142] Lustre: server umount lustre-MDT0001 complete [ 9145.416944] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9147.998606] Key type lgssc unregistered [ 9148.468408] LNet: 320184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9148.495068] LNetError: 320184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9149.543301] LNet: Removed LNI 192.168.203.155@tcp [ 9150.451392] Key type .llcrypt unregistered [ 9150.454712] Key type ._llcrypt unregistered [ 9173.820126] Key type ._llcrypt registered [ 9173.823384] Key type .llcrypt registered [ 9173.920770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9174.876843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9174.917073] alg: No test for adler32 (adler32-zlib) [ 9176.017596] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 9176.249576] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9177.922591] Key type lgssc registered [ 9179.088750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9186.826405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9186.856052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.377243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.754769] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9198.564790] LustreError: 321602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.580911] LustreError: 321602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9201.649913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.066935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9207.857840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9218.678929] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9226.282768] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9234.931401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9235.401241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.586616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9249.189667] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9268.705366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9268.714242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.734468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9269.227050] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9269.241886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9269.252033] Lustre: Skipped 2 previous similar messages [ 9271.503503] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 9274.341616] LustreError: 321598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9274.351945] Lustre: server umount lustre-MDT0000 complete [ 9274.367203] LustreError: 321598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9276.573777] LustreError: 324096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.312955] LustreError: 321582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151188 with bad export cookie 16926823832151807425 [ 9278.319524] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9278.326429] LustreError: 321582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9279.460967] LustreError: 321597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9279.462921] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.504838] Lustre: Skipped 1 previous similar message [ 9279.515581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9283.489936] LustreError: 321599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9283.509473] LustreError: 321599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9284.783808] Lustre: server umount lustre-MDT0001 complete [ 9290.896223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9299.448281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.035321] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xeae82897db067381 [ 9304.066659] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 9304.392898] LustreError: 323436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.480296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.568123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9309.492816] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9313.715364] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9318.641948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.883339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9318.896623] Lustre: Skipped 3 previous similar messages [ 9319.044494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9323.838221] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9324.522807] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9331.435517] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9332.851986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9338.495633] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9340.177909] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9345.508106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9345.513654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.526822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9349.603481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9349.614923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.518630] Lustre: server umount lustre-OST0000 complete [ 9361.314911] Lustre: server umount lustre-MDT0000 complete [ 9365.492111] LustreError: 325222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9365.508948] LustreError: 325222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9365.662579] LustreError: 321583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151275 with bad export cookie 16926823832151815041 [ 9365.663323] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9365.678516] LustreError: 321583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9366.111224] Lustre: server umount lustre-MDT0001 complete [ 9377.564353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9380.366782] Key type lgssc unregistered [ 9380.675724] LNet: 327897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9380.697753] LNetError: 327897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9380.736717] LNet: Removed LNI 192.168.203.155@tcp [ 9381.555794] Key type .llcrypt unregistered [ 9381.557412] Key type ._llcrypt unregistered [ 9409.393367] Key type ._llcrypt registered [ 9409.395589] Key type .llcrypt registered [ 9409.611958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9410.804976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9410.851251] alg: No test for adler32 (adler32-zlib) [ 9412.115799] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 9412.478500] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9414.240422] Key type lgssc registered [ 9415.538319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9423.778807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9423.808592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.455706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.674874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9435.629579] LustreError: 329330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.662697] LustreError: 329330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9439.216398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9439.548563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9443.969847] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9452.266379] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9459.293428] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.994092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.998181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9465.011366] Lustre: Skipped 1 previous similar message [ 9465.031743] Lustre: Skipped 1 previous similar message [ 9469.296666] Lustre: server umount lustre-MDT0000 complete [ 9470.453277] LustreError: 329330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9470.496392] LustreError: 329330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9473.597587] LustreError: 329310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151383 with bad export cookie 6917446443422422659 [ 9473.618515] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9474.326825] Lustre: server umount lustre-MDT0001 complete [ 9491.229773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.556792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9491.622969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9491.870120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.362529] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9504.764241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.855526] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9504.892943] Lustre: 331914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9504.978702] LustreError: 331927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9505.202429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9509.576588] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9516.003237] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9521.392793] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9530.276987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9530.407341] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9530.415183] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9530.423793] LustreError: 333800:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9530.429533] LustreError: 333800:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9530.433388] LustreError: 333800:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9530.497883] Lustre: server umount lustre-OST0000 complete [ 9530.506303] LustreError: 333800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9540.914608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.193602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9541.365692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9547.940415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9549.426150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9555.078796] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9563.572042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9563.693607] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9563.787112] Lustre: lustre-OST0001: new disk, initializing [ 9563.790789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9563.799758] Lustre: 335344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9563.881604] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9569.874423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9569.887383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9570.009749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9570.268037] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9577.086084] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9580.525138] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9580.534703] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9580.536544] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9580.545549] Lustre: Skipped 1 previous similar message [ 9580.560996] Lustre: Skipped 1 previous similar message [ 9585.649689] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9585.654925] Lustre: Skipped 1 previous similar message [ 9586.374417] Lustre: server umount lustre-OST0001 complete [ 9590.756881] LustreError: 334268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.758764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.779876] LustreError: 334268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9590.813025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9595.361277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9595.372432] LustreError: Skipped 1 previous similar message [ 9595.378495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.413170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9595.877411] LustreError: 334269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.909194] LustreError: 334269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9596.762607] Lustre: server umount lustre-OST0000 complete [ 9600.998615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.018992] Lustre: Skipped 1 previous similar message [ 9606.116361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9606.132285] Lustre: Skipped 4 previous similar messages [ 9606.781438] Lustre: server umount lustre-MDT0000 complete [ 9610.702695] LustreError: 331909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151520 with bad export cookie 6917446443422423212 [ 9610.710180] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.721856] LustreError: 331909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9611.186502] Lustre: server umount lustre-MDT0001 complete [ 9621.730556] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9624.492485] Key type lgssc unregistered [ 9624.908638] LNet: 337304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9624.916773] LNetError: 337304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9624.955235] LNet: Removed LNI 192.168.203.155@tcp [ 9625.950330] Key type .llcrypt unregistered [ 9625.954793] Key type ._llcrypt unregistered [ 9695.805492] Key type ._llcrypt registered [ 9695.807954] Key type .llcrypt registered [ 9695.916080] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9710.022154] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9711.302558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9711.312636] alg: No test for adler32 (adler32-zlib) [ 9712.343577] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [ 9712.581042] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9714.256272] Key type lgssc registered [ 9715.251873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9752.545123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9752.579456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.910733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9753.935190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9754.002377] Lustre: lustre-MDT0000: new disk, initializing [ 9754.094920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9754.111497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9758.249887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9769.747251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.008271] Lustre: 342248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9770.155197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9770.169714] Lustre: Skipped 1 previous similar message [ 9770.299241] Lustre: lustre-MDT0001: new disk, initializing [ 9770.444929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9770.484934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9770.504991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9774.965813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9786.368872] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9792.502742] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9799.766456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9800.041860] Lustre: lustre-OST0000: new disk, initializing [ 9800.044963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9800.051422] Lustre: 344217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9800.140113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9805.847501] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9807.967094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9807.978710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9808.093165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9815.702739] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9820.172165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9820.415142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9825.192628] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9825.450092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9828.836056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.852248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9828.862502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9828.869991] Lustre: Skipped 1 previous similar message [ 9833.786069] Lustre: server umount lustre-OST0000 complete [ 9839.074132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9839.076805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9839.087997] Lustre: Skipped 2 previous similar messages [ 9839.101438] Lustre: Skipped 1 previous similar message [ 9844.267631] Lustre: server umount lustre-MDT0000 complete [ 9848.284394] LustreError: 342241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151758 with bad export cookie 10165949119937851298 [ 9848.288606] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.293486] LustreError: 342241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9848.792800] Lustre: server umount lustre-MDT0001 complete [ 9857.946501] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:02:45 (1787151765) [ 9865.469488] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:02:53 (1787151773) [ 9873.178709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9873.600330] LustreError: 346973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9873.617871] LustreError: 346973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9873.691732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.870374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9879.012508] LustreError: 346974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.112311] LustreError: 346973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.651100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.161551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9891.483168] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9900.737722] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9907.252197] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9913.987933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9914.300654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9920.743946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9929.382689] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9940.820642] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9942.870722] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9944.875537] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9946.783928] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9949.065683] Lustre: Failing over lustre-MDT0000 [ 9949.551658] Lustre: server umount lustre-MDT0000 complete [ 9951.927635] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.241156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.268862] Lustre: Skipped 1 previous similar message [ 9957.011919] LustreError: 349462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.033120] LustreError: 349462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9967.254189] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.272913] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9968.519484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9968.608301] Lustre: 340463:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787151862/real 1787151862] req@0000000028c89a7e x1873964299263360/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787151878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9968.641599] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9968.954810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9969.030753] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9972.373886] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9973.701955] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9974.274960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9974.317149] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9974.388462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9983.254581] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9985.038365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9988.754251] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9990.673129] Lustre: Failing over lustre-MDT0000 [ 9990.869349] Lustre: server umount lustre-MDT0000 complete [ 9992.897515] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9992.922899] LustreError: 346969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9994.727737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9994.731122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9994.748083] Lustre: Skipped 2 previous similar messages [10009.787029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10010.003318] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10010.327871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10010.392934] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10013.343845] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10015.345581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10015.735949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10015.744334] Lustre: Skipped 2 previous similar messages [10015.780930] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10015.850556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10025.411721] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10026.912320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10036.193625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.205562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10036.217234] Lustre: Skipped 2 previous similar messages [10036.240136] Lustre: Skipped 2 previous similar messages [10036.246066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10039.339708] Lustre: server umount lustre-MDT0000 complete [10041.319350] LustreError: 347996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10041.359043] LustreError: 347996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [10043.710933] LustreError: 346954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787151953 with bad export cookie 10165949119937854007 [10043.729879] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.166200] Lustre: server umount lustre-MDT0001 complete [10059.104910] Lustre: server umount lustre-OST0000 complete [10073.494541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.075658] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10086.602088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.952111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10086.958224] Lustre: Skipped 1 previous similar message [10091.555858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10098.809868] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10104.619128] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10111.115358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10116.983733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10118.651678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10126.080261] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10131.937589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10131.954369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.979114] Lustre: Skipped 1 previous similar message [10131.992066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10133.708768] Lustre: server umount lustre-OST0000 complete [10139.116431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.125973] Lustre: Skipped 1 previous similar message [10148.322459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10148.328393] Lustre: Skipped 3 previous similar messages [10148.422725] Lustre: server umount lustre-MDT0000 complete [10152.371391] LustreError: 353306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152062 with bad export cookie 10165949119937854756 [10152.386760] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.388826] LustreError: 353306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10152.864223] Lustre: server umount lustre-MDT0001 complete [10163.093050] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10165.538619] Key type lgssc unregistered [10165.781223] LNet: 356863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10165.789205] LNetError: 356863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10165.810219] LNet: Removed LNI 192.168.203.155@tcp [10166.706107] Key type .llcrypt unregistered [10166.709614] Key type ._llcrypt unregistered [10195.822543] Key type ._llcrypt registered [10195.826635] Key type .llcrypt registered [10195.912461] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10196.642669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10196.669362] alg: No test for adler32 (adler32-zlib) [10197.932787] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [10198.244893] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10199.864598] Key type lgssc registered [10200.818981] Lustre: Echo OBD driver; http://www.lustre.org/ [10208.769461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10208.789723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.361623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10214.155753] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10220.514611] LustreError: 358385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.535460] LustreError: 358385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10221.296858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.590947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10226.015895] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10232.712909] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10238.906656] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10245.891844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10246.170966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10251.942357] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10255.363572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10259.775610] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10270.693081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.711521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.720591] Lustre: Skipped 2 previous similar messages [10275.810622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.826797] Lustre: Skipped 1 previous similar message [10276.463696] Lustre: server umount lustre-MDT0000 complete [10280.640287] LustreError: 358366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152190 with bad export cookie 14631207897393340863 [10280.643418] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.655491] LustreError: 358366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10280.937472] LustreError: 359626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10280.957730] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10280.974703] Lustre: Skipped 2 previous similar messages [10281.184058] Lustre: server umount lustre-MDT0001 complete [10287.553538] Lustre: server umount lustre-OST0000 complete [10301.244449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.797469] LustreError: 362092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.830397] LustreError: 362092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10301.911851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10306.491051] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10307.042099] LustreError: 362093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10311.139292] LustreError: 362092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.338048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10315.774256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10320.278584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10328.684630] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10334.641855] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10341.441939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10341.684159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10346.797106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10348.251042] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10356.354714] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10370.532066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.540908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10370.573699] Lustre: Skipped 2 previous similar messages [10370.578309] Lustre: Skipped 1 previous similar message [10375.653315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.661641] Lustre: Skipped 1 previous similar message [10376.378474] Lustre: server umount lustre-MDT0000 complete [10379.532742] LustreError: 362074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152289 with bad export cookie 14631207897393341976 [10379.541676] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.545483] LustreError: 362074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10379.858267] Lustre: server umount lustre-MDT0001 complete [10393.350768] Lustre: server umount lustre-OST0000 complete [10408.834315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.293021] LustreError: 366005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.451228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10414.380559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10421.717701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.027344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10433.968754] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.115452] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10446.788097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.310346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.322382] Lustre: Skipped 1 previous similar message [10454.471177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10454.880567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10462.921637] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10469.861207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10469.885273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.907158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10473.815144] Lustre: server umount lustre-OST0000 complete [10478.561400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10478.573131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.581178] Lustre: Skipped 1 previous similar message [10478.587899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10478.594502] Lustre: Skipped 2 previous similar messages [10492.384314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10492.500497] Lustre: server umount lustre-MDT0000 complete [10496.485205] LustreError: 366732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.515523] LustreError: 366732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10496.949790] LustreError: 365984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152406 with bad export cookie 14631207897393342998 [10496.955560] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10496.979117] LustreError: 365984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10497.385691] Lustre: server umount lustre-MDT0001 complete [10510.382599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10513.229988] Key type lgssc unregistered [10513.503537] LNet: 369541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10513.510946] LNetError: 369541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10513.531712] LNet: Removed LNI 192.168.203.155@tcp [10514.385087] Key type .llcrypt unregistered [10514.386697] Key type ._llcrypt unregistered [10541.169618] Key type ._llcrypt registered [10541.171458] Key type .llcrypt registered [10541.271763] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10542.091047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10542.108780] alg: No test for adler32 (adler32-zlib) [10543.140296] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [10543.335327] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10544.976194] Key type lgssc registered [10546.389396] Lustre: Echo OBD driver; http://www.lustre.org/ [10556.448197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10556.486208] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10556.497572] LustreError: 370982:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10556.541869] LustreError: 370982:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10556.553460] LustreError: 370982:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10556.569540] LustreError: 370982:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10556.588159] LustreError: 370982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10565.163464] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10565.169454] LustreError: 371380:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10565.188840] LustreError: 371380:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10565.198133] LustreError: 371380:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10565.207596] LustreError: 371380:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10565.214057] LustreError: 371380:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10584.768244] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10587.356872] Key type lgssc unregistered [10587.583522] LNet: 372388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10587.588489] LNetError: 372388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10587.610732] LNet: Removed LNI 192.168.203.155@tcp [10588.352473] Key type .llcrypt unregistered [10588.355564] Key type ._llcrypt unregistered [10605.901746] Key type ._llcrypt registered [10605.905479] Key type .llcrypt registered [10606.048304] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10621.377344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10623.021971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10623.073453] alg: No test for adler32 (adler32-zlib) [10624.142762] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [10624.435343] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10626.123345] Key type lgssc registered [10627.262320] Lustre: Echo OBD driver; http://www.lustre.org/ [10663.016132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10663.048244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.238387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10664.284515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10664.352549] Lustre: lustre-MDT0000: new disk, initializing [10664.416062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10664.431755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10669.666479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10680.611804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10680.729657] Lustre: 375545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10680.806990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10680.814683] Lustre: Skipped 1 previous similar message [10680.922591] Lustre: lustre-MDT0001: new disk, initializing [10681.046148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10681.083980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10681.113355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10685.836679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10696.080681] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10701.748937] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10708.657211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10709.015314] Lustre: lustre-OST0000: new disk, initializing [10709.022508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10709.032969] Lustre: 377514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10709.132638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10714.751659] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10717.817283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10717.828566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10718.018563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10724.389937] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10728.489182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10728.665539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10732.591960] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10732.803034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10738.657062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10738.658354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10738.673282] LustreError: Skipped 1 previous similar message [10738.674444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10738.703262] Lustre: Skipped 1 previous similar message [10738.724993] Lustre: Skipped 1 previous similar message [10741.059880] Lustre: server umount lustre-OST0000 complete [10747.876108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10747.897591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.920342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10748.903562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.150746] Lustre: server umount lustre-MDT0000 complete [10754.621781] LustreError: 375537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152664 with bad export cookie 16374353831895347516 [10754.632821] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10754.635996] LustreError: 375537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10754.899212] Lustre: server umount lustre-MDT0001 complete [10762.833479] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10764.736691] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:17:52 (1787152672) [10771.859265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.510366] LustreError: 379975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.526930] LustreError: 379975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10772.582256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10777.603659] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10778.081917] LustreError: 379976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.190682] LustreError: 379975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.104951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10787.310413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10787.321251] Lustre: Skipped 2 previous similar messages [10787.498463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10791.919584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10800.548401] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10807.086385] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10814.905709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10815.300057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10823.230854] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10833.078243] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10840.817963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10841.066698] Lustre: lustre-OST0001: new disk, initializing [10841.072876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10841.086362] Lustre: 382811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10841.205027] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10848.380211] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10848.404807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10848.418064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10848.588048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10859.923630] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10864.113965] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.116658] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10864.122879] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10864.128399] Lustre: Skipped 1 previous similar message [10869.232763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10869.246623] Lustre: Skipped 3 previous similar messages [10870.000435] Lustre: server umount lustre-OST0001 complete [10871.963250] LustreError: 381798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10879.475376] LustreError: 381795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10879.500570] LustreError: 381795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10889.186130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10889.188579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.192097] LustreError: Skipped 2 previous similar messages [10889.200739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10889.219810] Lustre: Skipped 1 previous similar message [10889.245644] Lustre: Skipped 1 previous similar message [10889.698673] LustreError: 381898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10889.736634] LustreError: 381898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10893.382981] Lustre: server umount lustre-OST0000 complete [10899.941976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10899.946781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10899.976724] Lustre: Skipped 1 previous similar message [10903.731493] Lustre: server umount lustre-MDT0000 complete [10907.675531] LustreError: 379957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152817 with bad export cookie 16374353831895348398 [10907.679120] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10907.686697] LustreError: 379957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10913.967603] Lustre: server umount lustre-MDT0001 complete [10926.895772] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10929.861529] Key type lgssc unregistered [10930.218177] LNet: 384802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10930.232925] LNetError: 384802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10930.259515] LNet: Removed LNI 192.168.203.155@tcp [10931.036509] Key type .llcrypt unregistered [10931.042337] Key type ._llcrypt unregistered [10982.404765] Key type ._llcrypt registered [10982.410880] Key type .llcrypt registered [10982.551770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10983.522024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10983.646287] alg: No test for adler32 (adler32-zlib) [10984.823225] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [10985.167703] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10986.840153] Key type lgssc registered [10988.051538] Lustre: Echo OBD driver; http://www.lustre.org/ [10996.321052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10996.345853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.613855] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10997.637645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10997.785974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.904813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11011.731049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11011.878302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11011.927344] Lustre: 387604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11012.151140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11016.569190] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11024.488492] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11030.753895] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11033.057784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11033.073112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11038.177546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11038.185449] Lustre: Skipped 2 previous similar messages [11043.297414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11043.306286] Lustre: Skipped 1 previous similar message [11047.392552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11047.538717] Lustre: server umount lustre-MDT0000 complete [11050.747077] LustreError: 387595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787152960 with bad export cookie 7417088825269662626 [11050.761160] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11051.207153] Lustre: server umount lustre-MDT0001 complete [11061.021747] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:22:48 (1787152968) [11067.400472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.742259] LustreError: 390117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.762611] LustreError: 390117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11067.801144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11071.588674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11072.998730] LustreError: 390118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11078.120049] LustreError: 390117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.441333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.772428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11084.991976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11091.895225] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11097.060798] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11103.131325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.312086] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11103.529494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11110.230365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11117.621551] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11124.858221] Lustre: Permanently deactivating lustre-OST0000 [11124.892996] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11130.444662] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11132.687713] LustreError: 393056:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11132.695775] LustreError: 393056:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11132.702257] LustreError: 393056:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11136.619930] LustreError: 393104:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11136.630829] LustreError: 393104:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11145.063768] Lustre: server umount lustre-OST0000 complete [11148.499227] LustreError: 393354:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11148.507436] LustreError: 393354:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11152.353115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11152.356520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.368860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11152.380139] Lustre: Skipped 2 previous similar messages [11152.393309] Lustre: Skipped 1 previous similar message [11157.472924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11157.478313] Lustre: Skipped 1 previous similar message [11160.630045] Lustre: server umount lustre-MDT0000 complete [11162.596919] LustreError: 390843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11162.626518] LustreError: 390843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11164.280545] LustreError: 390097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153074 with bad export cookie 7417088825269663207 [11164.281880] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.292449] LustreError: 390097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11175.157666] Lustre: server umount lustre-MDT0001 complete [11183.504260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.807278] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11191.934033] LustreError: 394514:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11198.303607] Lustre: 394853:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [11200.533556] Lustre: server umount MGS complete [11212.171141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.022218] LustreError: 395578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11213.082903] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11213.102177] Lustre: Skipped 1 previous similar message [11213.197436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.603510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11223.526481] LustreError: 395579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11223.562214] LustreError: 395579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11231.545634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.926852] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11232.043316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11237.054394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11244.894719] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11251.488576] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11253.697576] Lustre: Permanently reactivating lustre-OST0000 [11253.707740] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11253.715312] Lustre: Skipped 2 previous similar messages [11281.889561] LustreError: 397359:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11289.085987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11289.524158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11296.011988] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11304.083520] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11309.905763] Lustre: server umount lustre-OST0000 complete [11314.154746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11314.160136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11314.175800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.226141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11323.478776] Lustre: server umount lustre-MDT0000 complete [11327.190613] LustreError: 395558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153237 with bad export cookie 7417088825269664257 [11327.197483] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.204838] LustreError: 395558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11337.846772] Lustre: server umount lustre-MDT0001 complete [11349.513872] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11352.714573] Key type lgssc unregistered [11353.079079] LNet: 399948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11353.085645] LNetError: 399948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11353.107638] LNet: Removed LNI 192.168.203.155@tcp [11353.909352] Key type .llcrypt unregistered [11353.911784] Key type ._llcrypt unregistered [11372.773121] Key type ._llcrypt registered [11372.778762] Key type .llcrypt registered [11372.897171] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11387.812451] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11388.930982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11389.060942] alg: No test for adler32 (adler32-zlib) [11390.143687] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [11390.414801] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11392.184156] Key type lgssc registered [11393.454893] Lustre: Echo OBD driver; http://www.lustre.org/ [11430.578625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11430.602263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.891864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11431.926534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11432.034073] Lustre: lustre-MDT0000: new disk, initializing [11432.131105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11432.149466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11436.386331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11447.266472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.434579] Lustre: 403105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11447.485490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11447.488395] Lustre: Skipped 1 previous similar message [11447.575809] Lustre: lustre-MDT0001: new disk, initializing [11447.702888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11447.761923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11447.777759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11452.492121] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11463.254367] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11469.092607] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11476.639812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.973759] Lustre: lustre-OST0000: new disk, initializing [11476.979427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11476.997610] Lustre: 405074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11477.130335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11483.249749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11486.809264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11486.823531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11486.946757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11493.564352] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11498.462783] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11498.742556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11503.513373] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11503.746449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11507.684477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11507.685407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11507.704659] LustreError: Skipped 1 previous similar message [11507.722898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11507.730340] Lustre: Skipped 1 previous similar message [11512.135093] Lustre: server umount lustre-OST0000 complete [11517.925653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.938912] Lustre: Skipped 1 previous similar message [11517.946063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11517.957474] Lustre: Skipped 1 previous similar message [11523.043729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11523.054787] Lustre: Skipped 1 previous similar message [11527.140097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.145604] Lustre: Skipped 1 previous similar message [11530.208131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11530.402484] Lustre: server umount lustre-MDT0000 complete [11534.435340] LustreError: 403096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153444 with bad export cookie 3574888167364752430 [11534.442149] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11534.456336] LustreError: 403096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11535.026854] Lustre: server umount lustre-MDT0001 complete [11542.918596] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11544.423732] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:30:52 (1787153452) [11550.889245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11551.281797] LustreError: 407537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.300879] LustreError: 407537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11551.331681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11556.321376] LustreError: 407536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.808475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11561.448633] LustreError: 407537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.368767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11566.562966] LustreError: 407536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.976397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11571.623771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11579.302478] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11584.549336] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11591.304681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11591.595490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11597.174139] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11604.572139] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11606.479863] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11608.998304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11617.250139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11617.257737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11617.263399] Lustre: Skipped 2 previous similar messages [11617.272634] Lustre: Skipped 1 previous similar message [11620.292112] Lustre: server umount lustre-OST0000 complete [11627.494057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11627.498707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11627.508374] Lustre: Skipped 1 previous similar message [11630.320932] Lustre: server umount lustre-MDT0000 complete [11632.609872] LustreError: 407536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11632.626605] LustreError: 407536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11633.626163] LustreError: 407516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153543 with bad export cookie 3574888167364753312 [11633.631185] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11633.649395] LustreError: 407516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11633.942969] Lustre: server umount lustre-MDT0001 complete [11647.520800] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11650.853648] Key type lgssc unregistered [11651.209753] LNet: 411266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11651.217339] LNetError: 411266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11651.230748] LNet: Removed LNI 192.168.203.155@tcp [11652.094365] Key type .llcrypt unregistered [11652.096199] Key type ._llcrypt unregistered [11677.391868] Key type ._llcrypt registered [11677.394392] Key type .llcrypt registered [11677.491883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11677.816382] Key type .llcrypt unregistered [11677.822100] Key type ._llcrypt unregistered [11699.797746] Key type ._llcrypt registered [11699.799732] Key type .llcrypt registered [11699.937253] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11701.203871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11701.220478] alg: No test for adler32 (adler32-zlib) [11702.657257] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [11703.094198] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11704.848631] Key type lgssc registered [11706.984587] Lustre: Echo OBD driver; http://www.lustre.org/ [11715.257291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11715.309495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.818945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11722.069808] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11726.306976] LustreError: 413485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11726.324488] LustreError: 413485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11731.061385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11731.429442] LustreError: 413484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.490297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11736.561639] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11736.599233] LustreError: 413479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11738.611393] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11741.672875] LustreError: 413485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.699780] LustreError: 413485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11741.712252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11745.751484] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11746.800540] LustreError: 413484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.819637] LustreError: 413484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11752.315146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11752.703941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11756.627833] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11767.033794] Lustre: server umount lustre-OST0000 complete [11772.901137] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11772.910807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11772.929572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11773.414185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11773.430551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11773.438846] Lustre: Skipped 1 previous similar message [11778.534076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11778.543016] Lustre: Skipped 1 previous similar message [11779.613599] Lustre: server umount lustre-MDT0000 complete [11782.825022] LustreError: 413464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153692 with bad export cookie 1955258797607236289 [11782.838025] LustreError: 413464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11782.840663] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11792.546878] Lustre: server umount lustre-MDT0001 complete [11803.605124] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11806.544614] Key type lgssc unregistered [11806.885946] LNet: 416636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11806.898212] LNetError: 416636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11806.933234] LNet: Removed LNI 192.168.203.155@tcp [11807.747859] Key type .llcrypt unregistered [11807.752262] Key type ._llcrypt unregistered [11833.177133] Key type ._llcrypt registered [11833.179577] Key type .llcrypt registered [11833.271786] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11834.469602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11834.498071] alg: No test for adler32 (adler32-zlib) [11835.614777] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [11835.921111] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11837.600233] Key type lgssc registered [11838.532752] Lustre: Echo OBD driver; http://www.lustre.org/ [11845.255298] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11845.282209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11860.832611] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11865.954135] LustreError: 418028:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [11892.576183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11898.256697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11904.860233] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11910.873161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11911.367699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11911.469753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11915.421599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11916.772613] LustreError: 418053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11916.789784] LustreError: 418053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11921.895839] LustreError: 418054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.726527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.105341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11929.234920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11937.115560] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11945.268822] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11952.609431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11952.619329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.635767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11953.124416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11953.144040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11953.164022] Lustre: Skipped 1 previous similar message [11954.619744] Lustre: server umount lustre-OST0000 complete [11959.777540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11959.792995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.820899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11960.554684] Lustre: server umount lustre-MDT0000 complete [11963.362517] LustreError: 419070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11963.363334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.746723] LustreError: 418035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153873 with bad export cookie 3187372920244416614 [11963.757787] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11963.768219] LustreError: 418035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11964.041494] Lustre: server umount lustre-MDT0001 complete [11973.990320] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11976.705537] Key type lgssc unregistered [11977.022488] LNet: 421603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11977.035537] LNetError: 421603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11977.053720] LNet: Removed LNI 192.168.203.155@tcp [11977.850544] Key type .llcrypt unregistered [11977.853572] Key type ._llcrypt unregistered [12000.605536] Key type ._llcrypt registered [12000.607386] Key type .llcrypt registered [12000.693590] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12001.644525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12001.798644] alg: No test for adler32 (adler32-zlib) [12002.869800] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [12003.042850] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12004.689041] Key type lgssc registered [12005.549692] Lustre: Echo OBD driver; http://www.lustre.org/ [12013.053818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12013.074388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.565219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12018.516925] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12024.806559] LustreError: 423019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12024.834648] LustreError: 423019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12026.927477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.517627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12031.910737] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12039.617731] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12044.992343] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12052.642512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12053.083477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12059.331286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12059.554595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12069.396629] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12074.992176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12074.996542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12075.007302] Lustre: Skipped 2 previous similar messages [12075.026735] Lustre: Skipped 1 previous similar message [12076.728402] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [12076.738657] Lustre: Skipped 1 previous similar message [12080.019599] Lustre: server umount lustre-MDT0000 complete [12080.101109] LustreError: 423019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.127916] LustreError: 423019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12081.812982] LustreError: 423016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.739613] LustreError: 423001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1787153992 with bad export cookie 11412529824109297847 [12084.661508] LustreError: 423000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787153994 with bad export cookie 11412529824109296951 [12084.665599] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.677775] LustreError: 423000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12085.217512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12085.222111] LustreError: 423747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.234928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12085.251204] LustreError: 423747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12085.258938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12089.251077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12089.251254] LustreError: 423015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.269446] LustreError: 423015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12090.881407] Lustre: server umount lustre-OST0000 complete [12097.316871] Lustre: server umount lustre-MDT0001 complete [12108.281665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12110.706829] Key type lgssc unregistered [12110.913678] LNet: 426708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12110.930616] LNetError: 426708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12110.946549] LNet: Removed LNI 192.168.203.155@tcp [12111.717580] Key type .llcrypt unregistered [12111.721385] Key type ._llcrypt unregistered [12136.906069] Key type ._llcrypt registered [12136.910807] Key type .llcrypt registered [12137.032364] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12138.086117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12138.239185] alg: No test for adler32 (adler32-zlib) [12139.347669] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [12139.522563] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12141.184471] Key type lgssc registered [12142.227590] Lustre: Echo OBD driver; http://www.lustre.org/ [12149.151755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12149.187544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12150.635180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12154.938879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12160.996449] LustreError: 428137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.026161] LustreError: 428137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12162.656519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12162.981025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12167.655433] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12174.874107] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12180.585340] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12187.898051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12188.172129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12194.736085] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12195.387888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12202.777515] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12207.651949] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12207.669382] LustreError: 428865:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a87e61e0 x1873966844112128/t0(0) o7->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 264/240 e 0 to 0 dl 0 ref 2 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12207.671375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12208.023670] Lustre: server umount lustre-MDT0001 complete [12212.722151] LustreError: 429716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.729320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.785500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12215.800628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.821640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12217.827089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12217.829324] LustreError: 428136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12217.880929] LustreError: 428136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12220.263180] Lustre: server umount lustre-OST0000 complete [12222.961073] LustreError: 428137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.868449] Lustre: server umount lustre-MDT0000 complete [12240.269319] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12242.855550] Key type lgssc unregistered [12243.178376] LNet: 431822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12243.190322] LNetError: 431822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12243.201790] LNet: Removed LNI 192.168.203.155@tcp [12244.280375] Key type .llcrypt unregistered [12244.284660] Key type ._llcrypt unregistered [12268.354656] Key type ._llcrypt registered [12268.358350] Key type .llcrypt registered [12268.501612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12283.661476] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12284.881877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12284.976270] alg: No test for adler32 (adler32-zlib) [12286.079387] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [12286.285589] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12287.960143] Key type lgssc registered [12289.097703] Lustre: Echo OBD driver; http://www.lustre.org/ [12330.719474] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12345.625388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12345.643026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12346.938040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12346.965354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12347.069750] Lustre: lustre-MDT0000: new disk, initializing [12347.168336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12347.203496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12352.824875] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12371.994683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12372.287705] Lustre: lustre-OST0000: new disk, initializing [12372.299072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12372.308911] Lustre: Skipped 1 previous similar message [12372.317114] Lustre: 436859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12372.418637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12378.878624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12381.316653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12381.325058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12381.369682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12388.630468] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12404.275503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.417671] Lustre: 435842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12404.559379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12404.710020] Lustre: lustre-MDT0001: new disk, initializing [12404.925469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12404.990387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12405.013937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12409.625825] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12418.806582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12419.067764] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12423.307404] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12423.513594] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12430.820883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12430.822745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12430.832882] Lustre: Skipped 1 previous similar message [12430.844872] Lustre: Skipped 1 previous similar message [12432.553216] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12433.988633] Lustre: server umount lustre-MDT0001 complete [12441.071298] LustreError: 435853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12441.090639] LustreError: 435853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12446.180243] LustreError: 435852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.206324] LustreError: 435852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12451.322482] LustreError: 437141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12451.350044] LustreError: 437141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12453.345230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12453.352581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.366634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12456.419658] LustreError: 435853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.421751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12458.938727] Lustre: server umount lustre-OST0000 complete [12461.537664] LustreError: 435852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.878853] Lustre: server umount lustre-MDT0000 complete [12482.370532] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12484.857969] Key type lgssc unregistered [12485.115741] LNet: 440610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12485.120932] LNetError: 440610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12485.131607] LNet: Removed LNI 192.168.203.155@tcp [12485.774528] Key type .llcrypt unregistered [12485.776640] Key type ._llcrypt unregistered [12519.809788] Key type ._llcrypt registered [12519.812667] Key type .llcrypt registered [12519.918797] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12520.171397] Key type .llcrypt unregistered [12520.173239] Key type ._llcrypt unregistered [12540.348242] Key type ._llcrypt registered [12540.353661] Key type .llcrypt registered [12540.546362] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12541.386303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12541.405861] alg: No test for adler32 (adler32-zlib) [12543.239474] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [12543.615360] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12545.288242] Key type lgssc registered [12546.738507] Lustre: Echo OBD driver; http://www.lustre.org/ [12575.672291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12575.705355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.965986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12577.007573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12577.137838] Lustre: lustre-MDT0000: new disk, initializing [12577.250366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12577.275844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12581.963331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12593.075914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12593.238549] Lustre: 443996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12593.303062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12593.306983] Lustre: Skipped 1 previous similar message [12593.428695] Lustre: lustre-MDT0001: new disk, initializing [12593.583358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12593.645492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12593.661207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12599.511862] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12610.878378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12616.251257] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12621.921035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12622.131898] Lustre: lustre-OST0000: new disk, initializing [12622.139169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12622.146656] Lustre: 445966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12622.227970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12628.021135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12628.033793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12628.129469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12628.289454] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12636.149894] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12643.812438] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.825568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12643.828304] Lustre: Skipped 1 previous similar message [12647.719072] Lustre: server umount lustre-MDT0000 complete [12650.959080] LustreError: 443990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787154560 with bad export cookie 13834693257186962423 [12650.964420] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12650.974073] LustreError: 443990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12651.206504] Lustre: server umount lustre-MDT0001 complete [12665.071251] Lustre: server umount lustre-OST0000 complete [12676.107275] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:49:43 (1787154583) [12682.530321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.916152] LustreError: 447993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.935342] LustreError: 447993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12683.000503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12687.389335] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12688.361127] LustreError: 447994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12692.453861] LustreError: 447993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.124494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.515977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12701.058256] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12709.202914] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12716.386800] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12736.957134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12737.104220] Lustre: Found index 0 for lustre-OST0000, updating log [12737.213326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12743.021549] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12751.036816] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12753.593446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12762.081430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12762.101340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12762.120967] Lustre: Skipped 1 previous similar message [12762.130370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12762.137979] Lustre: Skipped 2 previous similar messages [12763.177938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12763.192048] Lustre: Skipped 1 previous similar message [12767.839586] Lustre: server umount lustre-OST0000 complete [12788.706938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12788.723162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12788.752324] Lustre: Skipped 1 previous similar message [12788.758895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12792.663870] Lustre: server umount lustre-MDT0000 complete [12793.506053] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [12796.901615] LustreError: 447993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12796.938587] LustreError: 447993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12797.003459] LustreError: 449839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787154706 with bad export cookie 13834693257186963816 [12797.012187] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12797.381406] Lustre: server umount lustre-MDT0001 complete [12806.517474] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:51:54 (1787154714) [13428.630130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13429.226432] LustreError: 454420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13429.376845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13434.456266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13434.851943] LustreError: 454421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13439.971635] LustreError: 454420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13444.072638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13444.530625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13449.263343] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13457.439802] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13463.826352] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13484.706253] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13485.252750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13485.865634] Lustre: Found index 0 for lustre-OST0000, updating log [13486.300864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13494.688449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13503.969989] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13507.697411] Lustre: server umount lustre-OST0000 complete [13513.255417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13513.263566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13513.283254] Lustre: Skipped 2 previous similar messages [13513.308303] Lustre: Skipped 7 previous similar messages [13514.357659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13514.377392] Lustre: Skipped 1 previous similar message [13515.426406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13515.438044] Lustre: Skipped 1 previous similar message [13517.480456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13517.483444] Lustre: Skipped 2 previous similar messages [13518.751962] Lustre: server umount lustre-MDT0000 complete [13519.534336] LustreError: 454417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13519.549418] LustreError: 454417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13522.400351] LustreError: 454400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787155432 with bad export cookie 13834693257186964901 [13522.410796] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13522.413857] LustreError: 454400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13522.745084] Lustre: server umount lustre-MDT0001 complete [13544.100321] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13554.273506] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13567.281526] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13605.648826] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:05:13 (1787155513) [13627.747915] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13632.098726] Key type lgssc unregistered [13632.483715] LNet: 461048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13632.497239] LNetError: 461048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13747.241686] LNet: Removed LNI 192.168.203.155@tcp [13748.228947] Key type .llcrypt unregistered [13748.234172] Key type ._llcrypt unregistered [13766.587396] Key type ._llcrypt registered [13766.592657] Key type .llcrypt registered [13766.785681] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13768.357229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13768.392736] alg: No test for adler32 (adler32-zlib) [13769.578627] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [13769.838506] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13771.546606] Key type lgssc registered [13772.831566] Lustre: Echo OBD driver; http://www.lustre.org/ [13777.173975] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [13777.951909] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [13778.846439] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [13779.872047] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [13780.755189] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [13781.610646] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [13782.458454] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [13783.394403] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [13784.334567] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [13785.336552] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [13786.155417] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [13787.127910] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [13788.011310] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [13788.761631] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [13789.620390] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [13790.593322] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [13791.402313] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [13792.419715] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [13793.423044] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [13794.371473] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [13795.317678] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [13796.271153] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [13797.231305] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [13798.348866] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [13799.419546] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [13800.482458] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [13801.594749] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [13802.570617] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [13803.548335] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [13804.546102] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [13805.488693] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [13806.415963] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [13807.579757] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [13808.419132] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [13809.350984] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [13810.184883] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [13817.434785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13817.470689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13818.610125] Lustre: 464081:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13818.674411] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13818.685808] LNetError: Protocol error connecting to 192.168.203.155@tcp113 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13818.698455] LNetError: Skipped 2 previous similar messages [13818.774277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13818.802370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13818.902777] Lustre: lustre-MDT0000: new disk, initializing [13819.043333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13819.058944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13819.689897] LNetError: 461542:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13819.706897] LNetError: 461542:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [13819.731567] LNetError: Protocol error connecting to 192.168.203.155@tcp at host 192.168.203.155:988: is it running a compatible version of Lustre? [13824.226093] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13834.484041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13834.601601] Lustre: 464100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13834.697646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13834.708636] Lustre: Skipped 1 previous similar message [13834.834300] Lustre: lustre-MDT0001: new disk, initializing [13834.873121] Lustre: 464902:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13834.884302] Lustre: 464902:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13835.025361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13835.079286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13835.090483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13839.886992] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13841.474507] Lustre: 465234:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13841.486086] Lustre: 465234:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13848.674452] LNetError: 461540:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13848.701053] LNetError: 461540:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [13848.708387] LNetError: Protocol error connecting to 192.168.203.155@tcp118 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13848.737475] LNetError: Skipped 2 previous similar messages [13849.578164] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13851.746579] LNetError: 461542:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13851.760805] LNetError: 461542:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13851.774170] LNetError: Protocol error connecting to 192.168.203.155@tcp116 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13851.786667] LNetError: Skipped 1 previous similar message [13855.847211] LNetError: 464111:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13855.864492] LNetError: Protocol error connecting to 192.168.203.155@tcp109 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13856.209887] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13864.037479] LNetError: 464104:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13864.062732] LNetError: 464104:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13864.077890] LNetError: Protocol error connecting to 192.168.203.155@tcp116 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13864.090481] LNetError: Skipped 4 previous similar messages [13864.114847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13864.375826] Lustre: lustre-OST0000: new disk, initializing [13864.379614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13864.385265] Lustre: 466076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13864.480527] Lustre: 466081:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13864.485439] Lustre: 466081:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13864.528856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13871.298505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13873.196197] Lustre: 466519:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13873.209962] Lustre: 466519:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13873.319624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13873.345414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13873.467811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13881.444931] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13881.461541] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [13881.474322] LNetError: Protocol error connecting to 192.168.203.155@tcp109 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13881.491724] LNetError: Skipped 6 previous similar messages [13881.583861] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13894.116939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13894.123168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13894.135318] Lustre: Skipped 1 previous similar message [13894.152776] Lustre: Skipped 1 previous similar message [13899.244904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13899.258738] Lustre: Skipped 1 previous similar message [13903.328147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13903.574041] Lustre: server umount lustre-OST0000 complete [13909.473913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13909.489362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13909.498657] Lustre: Skipped 1 previous similar message [13913.929731] Lustre: server umount lustre-MDT0000 complete [13917.935743] LustreError: 464091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787155827 with bad export cookie 8854037979625638163 [13917.944896] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13917.949445] LustreError: 464091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13918.278805] Lustre: server umount lustre-MDT0001 complete [13922.404317] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13922.416026] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13922.426843] LNetError: Protocol error connecting to 192.168.203.155@tcp120 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13922.438931] LNetError: Skipped 10 previous similar messages [13930.733661] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13934.142842] Key type lgssc unregistered [13934.499128] LNet: 467991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13934.512226] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13934.521827] LNet: Removed LNI 192.168.203.155@tcp [13935.584534] LNet: Removed LNI 192.168.203.155@tcp100 [13935.600632] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13935.619508] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13935.631636] LNet: Removed LNI 192.168.203.155@tcp101 [13936.672155] LNet: Removed LNI 192.168.203.155@tcp102 [13936.674364] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13936.687774] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13936.694395] LNet: Removed LNI 192.168.203.155@tcp103 [13937.762047] LNet: Removed LNI 192.168.203.155@tcp104 [13938.784201] LNet: Removed LNI 192.168.203.155@tcp105 [13938.791113] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13938.804775] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13938.811095] LNet: Removed LNI 192.168.203.155@tcp106 [13939.872187] LNet: Removed LNI 192.168.203.155@tcp107 [13940.898753] LNet: Removed LNI 192.168.203.155@tcp108 [13941.922763] LNet: Removed LNI 192.168.203.155@tcp109 [13942.944176] LNet: Removed LNI 192.168.203.155@tcp110 [13942.946134] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13942.952431] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13942.961153] LNet: Removed LNI 192.168.203.155@tcp111 [13943.968144] LNet: Removed LNI 192.168.203.155@tcp112 [13944.992164] LNet: Removed LNI 192.168.203.155@tcp113 [13946.016198] LNet: Removed LNI 192.168.203.155@tcp114 [13947.041042] LNet: Removed LNI 192.168.203.155@tcp115 [13948.064189] LNet: Removed LNI 192.168.203.155@tcp116 [13949.088288] LNet: Removed LNI 192.168.203.155@tcp117 [13950.112121] LNet: Removed LNI 192.168.203.155@tcp118 [13951.136153] LNet: Removed LNI 192.168.203.155@tcp119 [13951.144598] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13951.156373] LNetError: 467991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13951.165623] LNet: Removed LNI 192.168.203.155@tcp120 [13952.224413] LNet: Removed LNI 192.168.203.155@tcp121 [13952.228878] LNet: Removed LNI 192.168.203.155@tcp122 [13952.231441] LNet: Removed LNI 192.168.203.155@tcp123 [13952.240579] LNet: Removed LNI 192.168.203.155@tcp124 [13952.243340] LNet: Removed LNI 192.168.203.155@tcp125 [13952.246625] LNet: Removed LNI 192.168.203.155@tcp126 [13952.253143] LNet: Removed LNI 192.168.203.155@tcp127 [13952.258257] LNet: Removed LNI 192.168.203.155@tcp128 [13952.262111] LNet: Removed LNI 192.168.203.155@tcp129 [13952.268489] LNet: Removed LNI 192.168.203.155@tcp130 [13952.272450] LNet: Removed LNI 192.168.203.155@tcp131 [13952.276272] LNet: Removed LNI 192.168.203.155@tcp132 [13952.279363] LNet: Removed LNI 192.168.203.155@tcp133 [13952.283734] LNet: Removed LNI 192.168.203.155@tcp134 [13952.297995] LNet: Removed LNI 192.168.203.155@tcp135 [13952.961099] Key type .llcrypt unregistered [13952.966521] Key type ._llcrypt unregistered [13980.311954] Key type ._llcrypt registered [13980.314974] Key type .llcrypt registered [13980.427973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13996.739757] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13997.788125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13997.817873] alg: No test for adler32 (adler32-zlib) [13998.994975] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [13999.362436] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14001.072166] Key type lgssc registered [14002.335511] Lustre: Echo OBD driver; http://www.lustre.org/ [14040.935853] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:12:28 (1787155948) [14059.660447] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14062.688562] Key type lgssc unregistered [14062.921596] LNet: 472083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14062.927958] LNetError: 472083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14062.941975] LNet: Removed LNI 192.168.203.155@tcp [14063.852490] Key type .llcrypt unregistered [14063.857904] Key type ._llcrypt unregistered [14081.236189] Key type ._llcrypt registered [14081.241056] Key type .llcrypt registered [14081.397631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14082.402485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14082.549027] alg: No test for adler32 (adler32-zlib) [14083.793757] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [14084.109710] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14085.792262] Key type lgssc registered [14087.143647] Lustre: Echo OBD driver; http://www.lustre.org/ [14097.258333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14097.292058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14098.513898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14098.542715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14098.601756] Lustre: lustre-MDT0000: new disk, initializing [14098.651116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14098.662294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14103.119559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14113.806964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14113.938354] Lustre: 473296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14114.018882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14114.023304] Lustre: Skipped 1 previous similar message [14114.120277] Lustre: lustre-MDT0001: new disk, initializing [14114.238971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14114.283766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14114.301743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14118.735294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14129.375537] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14135.763735] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14142.517386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14142.755582] Lustre: lustre-OST0000: new disk, initializing [14142.758920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14142.764324] Lustre: 475265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14142.838817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14148.717785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14148.742126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14148.885381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14150.031489] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14159.563788] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14164.068794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [14169.548266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14169.586898] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [14174.745336] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14174.777568] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [14179.051685] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14179.100765] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [14183.620065] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14183.663743] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [14188.040613] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14188.062421] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [14193.004745] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14193.029499] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [14196.930948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14196.949478] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [14201.944238] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14201.986115] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [14209.269856] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14209.351858] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [14217.710946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14217.748948] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [14224.229457] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14224.286850] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [14229.915668] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14229.959601] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [14235.976525] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14236.007348] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [14242.689890] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14242.734321] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [14248.606869] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14248.670499] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [14254.968447] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14254.998475] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [14261.180287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14261.216782] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [14267.818897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14267.844978] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [14274.338269] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14274.392263] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [14281.425793] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14281.492680] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [14289.229190] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14289.257058] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [14296.695964] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14296.746307] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [14304.435849] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14304.457693] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [14311.710182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14311.765984] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [14320.256425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14320.306078] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [14328.605716] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14328.652234] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [14337.173544] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14337.220513] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [14345.597711] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14345.638876] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [14353.905371] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14353.944715] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [14362.650971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14362.687965] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [14371.924216] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14371.976035] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [14380.856545] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14380.898346] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [14388.921801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14388.950505] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [14397.260514] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14397.306980] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [14405.256562] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14405.300823] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [14414.683899] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14414.727407] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [14422.972938] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14423.001572] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [14430.859139] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14430.905520] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [14439.660586] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14439.706733] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [14447.435845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14447.462446] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [14455.198160] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14455.232750] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [14476.780375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14476.797234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14476.805580] Lustre: Skipped 2 previous similar messages [14476.833628] Lustre: Skipped 2 previous similar messages [14481.013437] Lustre: server umount lustre-MDT0000 complete [14485.035720] LustreError: 473287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156394 with bad export cookie 11006622374983577906 [14485.050888] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14485.060832] LustreError: 473287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14485.662239] Lustre: server umount lustre-MDT0001 complete [14502.369446] Lustre: 472571:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156396/real 1787156396] req@000000001b75bed9 x1873968883738368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787156412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14502.437886] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14505.578250] Lustre: 472571:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156399/real 1787156399] req@0000000071529c65 x1873968883738496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787156415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14507.106225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14507.488222] Lustre: 472568:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156401/real 1787156401] req@000000009edc3984 x1873968883738624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787156417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14510.629322] Lustre: 472568:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156404/real 1787156404] req@000000006e9c2b6d x1873968883738880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787156420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14515.681121] Lustre: 472570:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156409/real 1787156409] req@00000000f72161ff x1873968883739008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787156425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14516.808750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14529.427907] LustreError: 475259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14529.462179] LustreError: 475259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14529.516515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14531.563617] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14531.577759] LustreError: 475260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14533.601099] LustreError: 475259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14533.783358] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14538.722877] LustreError: 475260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14542.385978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14542.818161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14547.881599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14548.994076] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14556.604014] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14562.248415] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14588.905890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14588.911377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14588.924270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14589.930265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14589.937529] Lustre: Skipped 1 previous similar message [14592.922928] Lustre: server umount lustre-OST0000 complete [14599.139260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14599.144498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14599.164065] Lustre: Skipped 1 previous similar message [14599.169428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14604.509405] Lustre: server umount lustre-MDT0000 complete [14605.284508] LustreError: 499510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14605.321556] LustreError: 499510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14608.112465] LustreError: 473287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156518 with bad export cookie 11006622374983580531 [14608.117043] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14608.125564] LustreError: 473287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14608.761925] Lustre: server umount lustre-MDT0001 complete [14620.801992] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14623.697406] Key type lgssc unregistered [14624.120655] LNet: 502317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14624.131154] LNetError: 502317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14624.143951] LNet: Removed LNI 192.168.203.155@tcp [14624.150415] LNet: Removed LNI 192.168.203.155@tcp10 [14624.155662] LNet: Removed LNI 192.168.203.155@tcp11 [14624.165289] LNet: Removed LNI 192.168.203.155@tcp12 [14624.170605] LNet: Removed LNI 192.168.203.155@tcp13 [14624.173514] LNet: Removed LNI 192.168.203.155@tcp14 [14624.177453] LNet: Removed LNI 192.168.203.155@tcp15 [14624.180507] LNet: Removed LNI 192.168.203.155@tcp16 [14624.184165] LNet: Removed LNI 192.168.203.155@tcp17 [14624.186987] LNet: Removed LNI 192.168.203.155@tcp18 [14624.189466] LNet: Removed LNI 192.168.203.155@tcp19 [14624.192183] LNet: Removed LNI 192.168.203.155@tcp20 [14624.194118] LNet: Removed LNI 192.168.203.155@tcp21 [14624.196867] LNet: Removed LNI 192.168.203.155@tcp22 [14624.199589] LNet: Removed LNI 192.168.203.155@tcp23 [14624.204417] LNet: Removed LNI 192.168.203.155@tcp24 [14624.210773] LNet: Removed LNI 192.168.203.155@tcp25 [14624.218607] LNet: Removed LNI 192.168.203.155@tcp26 [14624.224588] LNet: Removed LNI 192.168.203.155@tcp27 [14624.230660] LNet: Removed LNI 192.168.203.155@tcp28 [14624.235513] LNet: Removed LNI 192.168.203.155@tcp29 [14624.238324] LNet: Removed LNI 192.168.203.155@tcp30 [14624.241329] LNet: Removed LNI 192.168.203.155@tcp31 [14624.243381] LNet: Removed LNI 192.168.203.155@tcp32 [14624.245922] LNet: Removed LNI 192.168.203.155@tcp33 [14624.248595] LNet: Removed LNI 192.168.203.155@tcp34 [14624.251995] LNet: Removed LNI 192.168.203.155@tcp35 [14624.261113] LNet: Removed LNI 192.168.203.155@tcp36 [14624.267076] LNet: Removed LNI 192.168.203.155@tcp37 [14624.273749] LNet: Removed LNI 192.168.203.155@tcp38 [14624.278261] LNet: Removed LNI 192.168.203.155@tcp39 [14624.281579] LNet: Removed LNI 192.168.203.155@tcp40 [14624.289955] LNet: Removed LNI 192.168.203.155@tcp41 [14624.293277] LNet: Removed LNI 192.168.203.155@tcp42 [14624.295945] LNet: Removed LNI 192.168.203.155@tcp43 [14624.298441] LNet: Removed LNI 192.168.203.155@tcp44 [14624.301481] LNet: Removed LNI 192.168.203.155@tcp45 [14624.304795] LNet: Removed LNI 192.168.203.155@tcp46 [14624.307487] LNet: Removed LNI 192.168.203.155@tcp47 [14624.310292] LNet: Removed LNI 192.168.203.155@tcp48 [14624.312398] LNet: Removed LNI 192.168.203.155@tcp49 [14624.325171] LNet: Removed LNI 192.168.203.155@tcp50 [14625.120814] Key type .llcrypt unregistered [14625.123361] Key type ._llcrypt unregistered [14645.986453] Key type ._llcrypt registered [14645.989191] Key type .llcrypt registered [14646.119173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14660.770780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14661.629268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14661.760129] alg: No test for adler32 (adler32-zlib) [14662.841985] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [14663.125159] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14664.848201] Key type lgssc registered [14665.811444] Lustre: Echo OBD driver; http://www.lustre.org/ [14704.178837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14704.207219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14705.363827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14705.403330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14705.458882] Lustre: lustre-MDT0000: new disk, initializing [14705.525491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14705.540940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14709.263477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14720.962279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14721.091287] Lustre: 505515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14721.200426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14721.208400] Lustre: Skipped 1 previous similar message [14721.303456] Lustre: lustre-MDT0001: new disk, initializing [14721.406506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14721.436963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14721.450578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14725.168164] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14734.744908] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14740.219931] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14748.635338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14748.878909] Lustre: *** cfs_fail_loc=90f, val=0*** [14749.004181] Lustre: lustre-OST0000: new disk, initializing [14749.008619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14749.020954] Lustre: 507536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14749.134205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14756.043750] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14757.940382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14757.952580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14758.096742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14765.989385] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14773.733241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14773.738191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14773.739582] LustreError: Skipped 1 previous similar message [14773.743077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14773.760039] Lustre: Skipped 1 previous similar message [14773.794029] Lustre: Skipped 1 previous similar message [14778.852383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14778.863300] Lustre: Skipped 1 previous similar message [14779.272756] Lustre: server umount lustre-OST0000 complete [14783.970201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14783.972906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14783.995831] Lustre: Skipped 1 previous similar message [14784.015153] Lustre: Skipped 1 previous similar message [14789.089407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14789.097836] Lustre: Skipped 1 previous similar message [14794.209735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14794.218951] Lustre: Skipped 1 previous similar message [14798.304134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14798.648369] Lustre: server umount lustre-MDT0000 complete [14802.740584] LustreError: 505509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156712 with bad export cookie 12631417339009562751 [14802.745289] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14802.756924] LustreError: 505509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14803.072556] Lustre: server umount lustre-MDT0001 complete [14813.726411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14816.877842] Key type lgssc unregistered [14817.183793] LNet: 509445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14817.188656] LNetError: 509445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14818.222815] LNet: Removed LNI 192.168.203.155@tcp [14819.297174] Key type .llcrypt unregistered [14819.302078] Key type ._llcrypt unregistered [14843.290882] Key type ._llcrypt registered [14843.293114] Key type .llcrypt registered [14843.401600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14844.648625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14844.690161] alg: No test for adler32 (adler32-zlib) [14845.826497] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [14846.058257] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14847.808478] Key type lgssc registered [14848.988594] Lustre: Echo OBD driver; http://www.lustre.org/ [14857.948808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14857.975679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14859.720724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14865.461497] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14869.995657] LustreError: 510723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14870.036860] LustreError: 510723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14874.671065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14874.927483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14879.926507] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14887.699335] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14893.324837] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14899.849573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14900.258558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14906.869108] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14914.485968] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14929.648143] loop0: detected capacity change from 0 to 204800000 [14929.964444] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14929.985215] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14930.009733] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14936.542852] loop0: detected capacity change from 0 to 204800000 [14936.783712] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14936.798432] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14936.812660] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14942.385308] loop0: detected capacity change from 0 to 204800000 [14948.945738] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14949.156279] Lustre: fs15246-OST0000: new disk, initializing [14949.160882] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14949.166155] Lustre: 514316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14949.218728] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14955.188929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14965.655501] loop1: detected capacity change from 0 to 204800000 [14972.512215] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14972.579649] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14972.621923] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14972.670543] Lustre: fs15246-MDT0000: new disk, initializing [14972.794977] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14972.808702] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14972.865691] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14973.924978] Lustre: 515439:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000858cd03a x1873969681579008/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787156899 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14977.747949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14983.074295] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14983.080769] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14983.109315] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15001.578990] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15001.609658] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15006.145207] Lustre: server umount fs15246-MDT0000 complete [15006.705891] LustreError: 510717:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15011.818528] LustreError: 510719:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15015.579579] Lustre: server umount fs15246-OST0000 complete [15032.296509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15032.304182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15032.314723] LustreError: Skipped 1 previous similar message [15032.327760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15032.344569] Lustre: Skipped 1 previous similar message [15032.378767] Lustre: Skipped 1 previous similar message [15034.218542] Lustre: server umount lustre-OST0000 complete [15038.945540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15038.954256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15038.972168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15042.537726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15042.545430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15044.707746] Lustre: server umount lustre-MDT0000 complete [15047.654110] LustreError: 510722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15047.678971] LustreError: 510722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15049.175873] LustreError: 511457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156959 with bad export cookie 13393839758263416826 [15049.182502] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15049.185391] LustreError: 511457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15049.788747] Lustre: server umount lustre-MDT0001 complete [15062.808868] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15067.089246] Key type lgssc unregistered [15067.640801] LNet: 518540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15067.658831] LNetError: 518540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15067.690820] LNet: Removed LNI 192.168.203.155@tcp [15068.944965] Key type .llcrypt unregistered [15068.947743] Key type ._llcrypt unregistered [15091.783588] Key type ._llcrypt registered [15091.790776] Key type .llcrypt registered [15091.902700] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15092.991318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15093.003696] alg: No test for adler32 (adler32-zlib) [15094.085486] Lustre: Lustre: Build Version: 2.17.57_44_g2cf1ecc [15094.334894] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [15095.976312] Key type lgssc registered [15097.166798] Lustre: Echo OBD driver; http://www.lustre.org/ [15132.991791] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [15144.574607] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15178.689923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15178.714841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15179.964823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15180.007692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15180.093071] Lustre: lustre-MDT0000: new disk, initializing [15180.178280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15180.219060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15185.197500] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15197.057849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15197.255229] Lustre: 523007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15197.416154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15197.428345] Lustre: Skipped 1 previous similar message [15197.529390] Lustre: lustre-MDT0001: new disk, initializing [15197.676067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15197.750057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15197.782021] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15203.413889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15214.047799] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15219.715258] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15226.814118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15227.107476] Lustre: lustre-OST0000: new disk, initializing [15227.111760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15227.120339] Lustre: 524973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15227.212830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15232.783797] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15235.747441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15235.770319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15235.995072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15242.567185] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15248.536851] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15248.796176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15253.831322] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15254.242684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15256.549126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15256.549253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15256.561177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15256.581304] LustreError: Skipped 1 previous similar message [15256.631953] Lustre: Skipped 1 previous similar message [15261.671474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15261.689037] Lustre: Skipped 2 previous similar messages [15262.726675] Lustre: server umount lustre-OST0000 complete [15266.786473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15266.788262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15266.795650] Lustre: Skipped 1 previous similar message [15266.818291] Lustre: Skipped 1 previous similar message [15271.909423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15271.922671] Lustre: Skipped 1 previous similar message [15272.741934] Lustre: server umount lustre-MDT0000 complete [15276.462479] LustreError: 526314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787157186 with bad export cookie 14319571599056158191 [15276.463860] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15276.468869] LustreError: 526314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15276.832444] Lustre: server umount lustre-MDT0001 complete [15301.214767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [15312.116596] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15342.227697] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15081 sec ========= 12:34:10 (1787157250) [15343.822338] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:34:11 (1787157251) === [15347.299965] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:34:15 (1787157255) === [15385.697141] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15388.200369] Key type lgssc unregistered [15388.417129] LNet: 531625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15388.432856] LNetError: 531625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15388.448304] LNet: Removed LNI 192.168.203.155@tcp [15389.231886] Key type .llcrypt unregistered [15389.235191] Key type ._llcrypt unregistered