[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469769830 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003260] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008556] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.015080] *** VALIDATE selinux *** [ 0.023749] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027847] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031201] *** VALIDATE tmpfs *** [ 0.033223] *** VALIDATE proc *** [ 0.034238] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037222] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.043890] debug: unmapping init [mem 0xffffffff99c59000-0xffffffff99c60fff] [ 0.046251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047593] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048020] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052010] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055315] rcu: Hierarchical SRCU implementation. [ 0.057336] smp: Bringing up secondary CPUs ... [ 0.058611] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062795] smp: Brought up 1 node, 4 CPUs [ 0.064027] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147022] node 0 deferred pages initialised in 79ms [ 0.151172] devtmpfs: initialized [ 0.152238] x86/mm: Memory block size: 128MB [ 0.154776] gcov: version magic: 0x41383552 [ 0.156139] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158228] pinctrl core: initialized pinctrl subsystem [ 0.159217] [ 0.159826] ************************************************************* [ 0.160017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161010] ** ** [ 0.162012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163009] ** ** [ 0.164014] ** This means that this kernel is built to expose internal ** [ 0.165013] ** IOMMU data structures, which may compromise security on ** [ 0.166011] ** your system. ** [ 0.167013] ** ** [ 0.168012] ** If you see this message and you are not debugging the ** [ 0.169014] ** kernel, report this immediately to your vendor! ** [ 0.170011] ** ** [ 0.171011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172012] ************************************************************* [ 0.173666] NET: Registered protocol family 16 [ 0.174407] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177444] cpuidle: using governor menu [ 0.179754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182514] PCI: Using configuration type 1 for base access [ 0.184127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200078] cryptd: max_cpu_qlen set to 1000 [ 0.204165] ACPI: Added _OSI(Module Device) [ 0.206015] ACPI: Added _OSI(Processor Device) [ 0.207011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213975] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221025] ACPI: Interpreter enabled [ 0.222062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224012] ACPI: Using IOAPIC for interrupt routing [ 0.226126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.230375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253404] acpiphp: Slot [2] registered [ 0.255132] acpiphp: Slot [5] registered [ 0.257181] acpiphp: Slot [6] registered [ 0.259155] acpiphp: Slot [7] registered [ 0.261182] acpiphp: Slot [8] registered [ 0.263135] acpiphp: Slot [9] registered [ 0.264200] acpiphp: Slot [10] registered [ 0.266117] acpiphp: Slot [3] registered [ 0.268109] acpiphp: Slot [4] registered [ 0.269103] acpiphp: Slot [11] registered [ 0.271160] acpiphp: Slot [12] registered [ 0.273097] acpiphp: Slot [13] registered [ 0.274103] acpiphp: Slot [14] registered [ 0.276219] acpiphp: Slot [15] registered [ 0.277098] acpiphp: Slot [16] registered [ 0.279103] acpiphp: Slot [17] registered [ 0.281171] acpiphp: Slot [18] registered [ 0.282087] acpiphp: Slot [19] registered [ 0.283096] acpiphp: Slot [20] registered [ 0.285122] acpiphp: Slot [21] registered [ 0.287083] acpiphp: Slot [22] registered [ 0.288123] acpiphp: Slot [23] registered [ 0.290083] acpiphp: Slot [24] registered [ 0.291102] acpiphp: Slot [25] registered [ 0.293103] acpiphp: Slot [26] registered [ 0.294125] acpiphp: Slot [27] registered [ 0.296198] acpiphp: Slot [28] registered [ 0.298113] acpiphp: Slot [29] registered [ 0.299072] acpiphp: Slot [30] registered [ 0.301109] acpiphp: Slot [31] registered [ 0.303056] PCI host bridge to bus 0000:00 [ 0.304016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321100] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323270] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.333079] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.338011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347918] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.350855] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357791] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375965] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.426433] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.479085] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.487023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.498036] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.519021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.530144] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.545018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.551013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.567025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.580428] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.620024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.632564] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.640020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.647023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.693197] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.696516] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.700479] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.703457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.705344] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.712272] iommu: Default domain type: Passthrough [ 0.715613] SCSI subsystem initialized [ 0.717220] ACPI: bus type USB registered [ 0.719146] usbcore: registered new interface driver usbfs [ 0.721151] usbcore: registered new interface driver hub [ 0.723114] usbcore: registered new device driver usb [ 0.726229] pps_core: LinuxPPS API ver. 1 registered [ 0.728016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.732088] PTP clock support registered [ 0.734140] EDAC MC: Ver: 3.0.0 [ 0.737191] PCI: Using ACPI for IRQ routing [ 0.739901] NetLabel: Initializing [ 0.741011] NetLabel: domain hash size = 128 [ 0.743013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.745079] NetLabel: unlabeled traffic allowed by default [ 0.748214] vgaarb: loaded [ 0.750293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.752020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.759746] clocksource: Switched to clocksource kvm-clock [ 0.878762] VFS: Disk quotas dquot_6.6.0 [ 0.880682] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883505] *** VALIDATE ramfs *** [ 0.885187] *** VALIDATE hugetlbfs *** [ 0.887199] pnp: PnP ACPI init [ 0.890647] pnp: PnP ACPI: found 6 devices [ 0.908419] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912469] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.915192] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917526] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.920478] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.923181] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.926229] NET: Registered protocol family 2 [ 0.928683] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.933940] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.937871] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.942527] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.946182] TCP: Hash tables configured (established 65536 bind 65536) [ 0.949290] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.952714] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955869] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960934] NET: Registered protocol family 1 [ 0.964050] RPC: Registered named UNIX socket transport module. [ 0.966562] RPC: Registered udp transport module. [ 0.968540] RPC: Registered tcp transport module. [ 0.970522] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.973330] NET: Registered protocol family 44 [ 0.975277] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.977689] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.980186] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.982735] PCI: CLS 0 bytes, default 64 [ 0.984714] Unpacking initramfs... [ 2.383120] debug: unmapping init [mem 0xffff88c9fcc54000-0xffff88c9fffbffff] [ 2.386899] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.391472] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.394779] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.352233] Initialise system trusted keyrings [ 3.358718] Key type blacklist registered [ 3.363185] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.379501] zbud: loaded [ 3.383628] *** VALIDATE nfs *** [ 3.387193] *** VALIDATE nfs4 *** [ 3.389344] pstore: using deflate compression [ 3.396462] Platform Keyring initialized [ 3.695108] NET: Registered protocol family 38 [ 3.697331] Key type asymmetric registered [ 3.704909] Asymmetric key parser 'x509' registered [ 3.711311] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.722292] io scheduler mq-deadline registered [ 3.726651] io scheduler kyber registered [ 3.730715] io scheduler bfq registered [ 3.740374] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.753049] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.770190] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.776790] ACPI: Power Button [PWRF] [ 3.789253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.805366] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.836170] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.854307] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.890644] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.924021] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.960517] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.971309] Non-volatile memory driver v1.3 [ 3.977081] Linux agpgart interface v0.103 [ 4.051593] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 4.058718] vda: detected capacity change from 0 to 76685312 [ 4.110503] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.120706] vdb: detected capacity change from 0 to 1073741824 [ 4.168771] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.173347] vdc: detected capacity change from 0 to 2621440000 [ 4.212804] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.220527] vdd: detected capacity change from 0 to 2621440000 [ 4.262690] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.267730] vde: detected capacity change from 0 to 4294967296 [ 4.295794] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.300261] vdf: detected capacity change from 0 to 4294967296 [ 4.315101] libphy: Fixed MDIO Bus: probed [ 4.338735] usbcore: registered new interface driver usbserial_generic [ 4.345271] usbserial: USB Serial support registered for generic [ 4.350330] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.359486] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.362693] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.367398] mousedev: PS/2 mouse device common for all mice [ 4.375117] rtc_cmos 00:05: RTC can wake from S4 [ 4.382317] rtc_cmos 00:05: registered as rtc0 [ 4.389445] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.395594] intel_pstate: CPU model not supported [ 4.407695] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.417562] hid: raw HID events driver (C) Jiri Kosina [ 4.421145] usbcore: registered new interface driver usbhid [ 4.425984] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.429682] usbhid: USB HID core driver [ 4.433274] drop_monitor: Initializing network drop monitor service [ 4.452534] Initializing XFRM netlink socket [ 4.455821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.461249] NET: Registered protocol family 10 [ 4.490835] Segment Routing with IPv6 [ 4.493435] NET: Registered protocol family 17 [ 4.504575] mpls_gso: MPLS GSO support [ 4.524181] RAS: Correctable Errors collector initialized. [ 4.530562] AVX version of gcm_enc/dec engaged. [ 4.533365] AES CTR mode by8 optimization enabled [ 4.836593] sched_clock: Marking stable (4836567963, 0)->(5778553017, -941985054) [ 4.844379] registered taskstats version 1 [ 4.863406] Loading compiled-in X.509 certificates [ 4.869948] zswap: loaded using pool lzo/zbud [ 4.977496] Key type big_key registered [ 5.031952] Key type encrypted registered [ 5.035551] ima: No TPM chip found, activating TPM-bypass! [ 5.040390] ima: Allocated hash algorithm: sha1 [ 5.045247] ima: No architecture policies found [ 5.049164] evm: Initialising EVM extended attributes: [ 5.051756] evm: security.selinux [ 5.053765] evm: security.ima [ 5.057810] evm: security.capability [ 5.065539] evm: HMAC attrs: 0x1 [ 5.079335] rtc_cmos 00:05: setting system clock to 2026-09-05 05:29:02 UTC (1788586142) [ 5.091712] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 5.101587] debug: unmapping init [mem 0xffffffff99982000-0xffffffff99c58fff] [ 5.124631] Write protecting the kernel read-only data: 28672k [ 5.131317] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 5.138934] debug: unmapping init [mem 0xffffffff98914000-0xffffffff989fffff] [ 5.338742] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.365614] systemd[1]: Detected virtualization kvm. [ 5.367690] systemd[1]: Detected architecture x86-64. [ 5.374341] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.480901] systemd[1]: No hostname configured. [ 5.486130] systemd[1]: Set hostname to . [ 5.493227] random: systemd: uninitialized urandom read (16 bytes read) [ 5.499369] systemd[1]: Initializing machine ID from random generator. [ 5.650871] random: ln: uninitialized urandom read (6 bytes read) [ 5.850212] random: systemd: uninitialized urandom read (16 bytes read) [ 5.854148] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.863397] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.889717] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.550474] device-mapper: uevent: version 1.0.3 [ 7.555172] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.228763] random: fast init done [ 9.281190] virtio_net virtio0 ens2: renamed from eth0 [ 9.563758] scsi host0: ata_piix [ 9.647830] scsi host1: ata_piix [ 9.653463] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.663505] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.592127] random: crng init done [ 14.597982] random: 7 urandom warning(s) missed due to ratelimiting [ 15.817511] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.663062] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.655282] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.431969] SELinux: Disabled at runtime. [ 21.544526] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.568419] systemd[1]: Detected virtualization kvm. [ 21.573122] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.269389] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.273966] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.284224] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.290511] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.302243] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.321112] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.352283] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ 23.832539] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 24.964634] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.283382] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.297455] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.454466] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.565408] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.158592] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.863957] NFS: Registering the id_resolver key type [ 31.866432] Key type id_resolver registered [ 31.868352] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 69.576319] hrtimer: interrupt took 4310344 ns [ 87.650671] libcfs: loading out-of-tree module taints kernel. [ 87.720393] Key type ._llcrypt registered [ 87.723822] Key type .llcrypt registered [ 87.879851] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 111.252942] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 112.930025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.972660] alg: No test for adler32 (adler32-zlib) [ 114.465298] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 115.232988] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 116.985938] Key type lgssc registered [ 118.616910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.970499] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.786570] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 193.872558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.939269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.247918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.294888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.398407] Lustre: lustre-MDT0000: new disk, initializing [ 195.454243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.473775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.497680] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.748295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.848391] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.880519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.885383] Lustre: Skipped 1 previous similar message [ 215.951195] Lustre: lustre-MDT0001: new disk, initializing [ 216.020412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.054552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.069884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.556536] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.274329] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.160804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.527350] Lustre: lustre-OST0000: new disk, initializing [ 234.532711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.540434] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.630739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.875791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.895356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.948124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.003530] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.704977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.823549] Lustre: lustre-OST0001: new disk, initializing [ 254.827454] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.832938] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.882043] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.428426] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.170320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.178241] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.242828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.171702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.783997] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 281.702061] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 287.411706] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 293.185904] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 295.888418] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 298.319769] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 300.778886] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 01:33:55 EDT 2026 [ 316.210555] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 317.493704] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.772026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.798222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.810282] Lustre: Skipped 3 previous similar messages [ 337.889984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.899469] Lustre: Skipped 3 previous similar messages [ 343.017470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.032350] Lustre: Skipped 3 previous similar messages [ 347.112808] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 347.356333] Lustre: server umount lustre-MDT0000 complete [ 350.681099] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586488 with bad export cookie 15501372903219424885 [ 350.686633] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.710991] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 353.252701] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.254209] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.266225] Lustre: Skipped 3 previous similar messages [ 353.270295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.280156] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 353.286200] Lustre: Skipped 1 previous similar message [ 355.297341] LustreError: 10421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.320822] LustreError: 10421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 357.070344] Lustre: server umount lustre-MDT0001 complete [ 360.912744] Lustre: server umount lustre-OST0000 complete [ 364.509575] Lustre: server umount lustre-OST0001 complete [ 370.132490] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 378.843800] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 410.845509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.013603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.033801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.089411] Lustre: lustre-MDT0000: new disk, initializing [ 411.156807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.179260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.502470] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 425.244908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.319267] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.381811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.385133] Lustre: Skipped 1 previous similar message [ 425.465896] Lustre: lustre-MDT0001: new disk, initializing [ 425.560759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 425.587585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 425.603963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.524409] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 437.174607] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.705858] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.693329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.039886] Lustre: lustre-OST0000: new disk, initializing [ 448.048084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.057492] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.163724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.639527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.644805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.774826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.538713] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 462.229434] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.251388] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 466.437395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.080580] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.249420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.620343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.622795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.651666] Lustre: Skipped 1 previous similar message [ 475.661078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.674736] Lustre: Skipped 2 previous similar messages [ 485.872165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.882578] Lustre: Skipped 3 previous similar messages [ 487.392252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 487.626994] Lustre: server umount lustre-OST0000 complete [ 492.003349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.009133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.101726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.498612] Lustre: server umount lustre-MDT0000 complete [ 500.667847] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586638 with bad export cookie 15501372903219426418 [ 500.674098] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.681284] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.897551] Lustre: server umount lustre-MDT0001 complete [ 505.107719] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:37:20 (1788586640) [ 507.754324] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 509.907474] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:37:24 (1788586644) [ 516.181623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.418923] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.486824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.583286] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 521.700015] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.797641] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.581284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.732739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 544.751392] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 550.857866] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.767362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 564.331518] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 570.790658] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 580.065603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 580.070705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.080496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 580.087312] Lustre: Skipped 3 previous similar messages [ 582.119417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.822314] Lustre: server umount lustre-OST0000 complete [ 592.764849] Lustre: server umount lustre-MDT0000 complete [ 595.987735] LustreError: 20189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586733 with bad export cookie 15501372903219427300 [ 595.989603] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.007288] LustreError: 20189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 596.260557] Lustre: server umount lustre-MDT0001 complete [ 607.552442] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 610.350979] Key type lgssc unregistered [ 610.664520] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.672802] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.694720] LNet: Removed LNI 192.168.203.150@tcp [ 611.552155] Key type .llcrypt unregistered [ 611.556799] Key type ._llcrypt unregistered [ 630.642450] Key type ._llcrypt registered [ 630.646299] Key type .llcrypt registered [ 630.739294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 631.513325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 631.532511] alg: No test for adler32 (adler32-zlib) [ 632.500615] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 632.680757] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 634.296169] Key type lgssc registered [ 635.239622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 642.063034] loop: module loaded [ 642.116433] loop0: detected capacity change from 0 to 204800000 [ 642.274318] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.289116] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.703837] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.000371] loop0: detected capacity change from 0 to 204800000 [ 648.167281] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.177786] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.191992] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.935975] loop0: detected capacity change from 0 to 204800000 [ 656.556890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.583079] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.805553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 657.868911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 657.955697] Lustre: lustre-MDT0000: new disk, initializing [ 658.030943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.052651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 661.662206] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 669.997409] loop1: detected capacity change from 0 to 204800000 [ 675.366872] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 675.773300] Lustre: lustre-OST07c6: new disk, initializing [ 675.783645] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 675.792659] Lustre: Skipped 1 previous similar message [ 675.797573] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 675.805853] Lustre: 26168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 675.923591] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 681.025939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 681.041893] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 681.076052] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 682.865919] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 688.816838] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 688.832474] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 688.842982] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 688.847943] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 688.865116] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 688.875432] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 706.700421] Lustre: 26849:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 706.712310] Lustre: 26849:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 706.718972] Lustre: 26849:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.723463] Lustre: 26849:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 706.728325] Lustre: 26849:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 706.733384] Lustre: 26849:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.582231] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 711.594685] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 711.605081] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 711.614531] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.626167] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.636213] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.643692] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 711.651688] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.664053] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.680149] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.690089] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.702207] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.977747] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 714.991065] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 714.999020] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 715.008269] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.021830] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 715.027973] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.037317] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 715.044777] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.049712] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 715.054324] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.058686] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 715.070306] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.681794] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 719.690420] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.700758] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 719.706801] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.711972] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.722263] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.735023] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 719.749120] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.759449] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 719.765529] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.772745] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.780988] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.677288] Lustre: 26850:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 730.685899] Lustre: 26850:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 730.690275] Lustre: 26850:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 730.695609] Lustre: 26850:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 730.700084] Lustre: 26850:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.703967] Lustre: 26850:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 730.709425] Lustre: 26850:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 730.712760] Lustre: 26850:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 730.716902] Lustre: 26850:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 730.720265] Lustre: 26850:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 730.723991] Lustre: 26850:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.729090] Lustre: 26850:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 778.211842] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 778.221761] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.247577] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 778.947338] Lustre: server umount lustre-OST07c6 complete [ 787.436900] Lustre: server umount lustre-MDT0000 complete [ 802.577371] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 805.345842] Key type lgssc unregistered [ 805.718747] LNet: 28721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.731826] LNetError: 28721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.759776] LNet: Removed LNI 192.168.203.150@tcp [ 806.695068] Key type .llcrypt unregistered [ 806.697959] Key type ._llcrypt unregistered [ 823.793320] Key type ._llcrypt registered [ 823.796131] Key type .llcrypt registered [ 823.923199] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 825.063694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.091877] alg: No test for adler32 (adler32-zlib) [ 826.156608] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 826.360596] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 828.008205] Key type lgssc registered [ 829.189856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 836.468621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 836.486709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.817213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.818867] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 848.363817] LustreError: 29984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.377158] LustreError: 29984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 849.552247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.906403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 853.959677] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 860.720288] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 865.308646] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 871.319124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.665198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.003583] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 882.939526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 884.126303] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 898.531739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.534831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.574951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.582518] Lustre: Skipped 1 previous similar message [ 903.686644] Lustre: server umount lustre-OST0000 complete [ 908.771164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.774478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.784557] Lustre: Skipped 2 previous similar messages [ 908.799437] Lustre: Skipped 1 previous similar message [ 913.792890] Lustre: server umount lustre-MDT0000 complete [ 913.889109] LustreError: 29985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.921710] LustreError: 29985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 917.693760] LustreError: 29965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587055 with bad export cookie 6569902323547115152 [ 917.710563] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.214981] Lustre: server umount lustre-MDT0001 complete [ 928.820907] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 931.811367] Key type lgssc unregistered [ 932.117536] LNet: 33722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.126906] LNetError: 33722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.141886] LNet: Removed LNI 192.168.203.150@tcp [ 933.112327] Key type .llcrypt unregistered [ 933.119167] Key type ._llcrypt unregistered [ 952.230971] Key type ._llcrypt registered [ 952.236936] Key type .llcrypt registered [ 952.399364] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 953.799263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 953.827943] alg: No test for adler32 (adler32-zlib) [ 954.889144] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 955.122182] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 956.816453] Key type lgssc registered [ 958.230532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 965.752136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 965.792176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.258237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.624763] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 977.381538] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.406428] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 979.843700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.162452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.907597] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 992.382870] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 997.373474] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1003.301921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1003.659936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.604461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1011.913923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1016.594118] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1027.560814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.564781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.567059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1027.585884] Lustre: Skipped 1 previous similar message [ 1032.675115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.685758] Lustre: Skipped 2 previous similar messages [ 1033.803239] Lustre: server umount lustre-OST0000 complete [ 1037.797220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.821395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.829445] Lustre: Skipped 1 previous similar message [ 1042.914131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.932439] Lustre: Skipped 1 previous similar message [ 1043.261465] Lustre: server umount lustre-MDT0000 complete [ 1046.694845] LustreError: 34968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587184 with bad export cookie 6936818788432382609 [ 1046.709074] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.174399] Lustre: server umount lustre-MDT0001 complete [ 1059.742735] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1063.127356] Key type lgssc unregistered [ 1063.415550] LNet: 38520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.424458] LNetError: 38520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.444322] LNet: Removed LNI 192.168.203.150@tcp [ 1064.347250] Key type .llcrypt unregistered [ 1064.349166] Key type ._llcrypt unregistered [ 1086.300588] Key type ._llcrypt registered [ 1086.303841] Key type .llcrypt registered [ 1086.412822] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1087.288682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.456596] alg: No test for adler32 (adler32-zlib) [ 1088.793375] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1089.184835] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1090.904194] Key type lgssc registered [ 1092.267480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.676666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.692321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.142578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.486088] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1111.533330] LustreError: 39935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.549134] LustreError: 39935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1113.431111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.820311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1118.310180] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1124.945244] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1129.939717] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1136.243792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1136.602101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.397453] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1145.859162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1149.132360] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1156.069622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.074868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.088028] Lustre: Skipped 2 previous similar messages [ 1156.110118] Lustre: Skipped 2 previous similar messages [ 1156.827969] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1157.776878] Lustre: server umount lustre-MDT0000 complete [ 1160.969723] LustreError: 39915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587298 with bad export cookie 3672593243942691700 [ 1160.980955] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.190050] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.192621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.227190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1165.217477] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.217517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1165.243488] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1165.257382] Lustre: Skipped 1 previous similar message [ 1167.463866] Lustre: server umount lustre-MDT0001 complete [ 1187.688680] Lustre: server umount lustre-OST0000 complete [ 1201.407547] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1204.296569] Key type lgssc unregistered [ 1204.631108] LNet: 43790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.645737] LNetError: 43790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.669138] LNet: Removed LNI 192.168.203.150@tcp [ 1205.580629] Key type .llcrypt unregistered [ 1205.583341] Key type ._llcrypt unregistered [ 1229.514641] Key type ._llcrypt registered [ 1229.516895] Key type .llcrypt registered [ 1229.596477] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1230.479156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1230.583221] alg: No test for adler32 (adler32-zlib) [ 1231.720960] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1231.932427] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1233.608242] Key type lgssc registered [ 1234.622414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1241.172747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1241.205271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.782068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.738835] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1253.358071] LustreError: 45205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.391240] LustreError: 45205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1255.614744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.927623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1260.224181] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1267.666722] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.722853] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1278.854289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1279.126380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.186592] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1288.387747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1292.145363] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1298.924457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.932738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.944405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.959593] Lustre: Skipped 1 previous similar message [ 1300.220491] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1304.035973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1304.046925] Lustre: Skipped 1 previous similar message [ 1304.833334] Lustre: server umount lustre-OST0000 complete [ 1322.466437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.476131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.493890] Lustre: Skipped 1 previous similar message [ 1322.510955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.513616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.563195] Lustre: server umount lustre-MDT0000 complete [ 1328.678127] LustreError: 46235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.700329] LustreError: 46235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1332.370370] LustreError: 45186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587469 with bad export cookie 17423756487698750454 [ 1332.379264] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.775770] Lustre: server umount lustre-MDT0001 complete [ 1349.281020] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1351.988235] Key type lgssc unregistered [ 1352.344281] LNet: 48912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.350491] LNetError: 48912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.372857] LNet: Removed LNI 192.168.203.150@tcp [ 1353.385142] Key type .llcrypt unregistered [ 1353.387716] Key type ._llcrypt unregistered [ 1375.175573] Key type ._llcrypt registered [ 1375.177734] Key type .llcrypt registered [ 1375.258778] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1376.467038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.513028] alg: No test for adler32 (adler32-zlib) [ 1377.519548] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1377.788016] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1379.456179] Key type lgssc registered [ 1380.670310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1387.873023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1387.883636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.270640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.077950] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1398.758377] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.776883] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1401.811872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.103039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.501440] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1413.558927] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.470905] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1425.189421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.555772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.388574] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1433.772243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1437.861591] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.858924] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1446.039112] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1458.089886] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-05 1h53m12s [ 1459.422123] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1463.784860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1463.799582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.825616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.871032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.879125] Lustre: Skipped 4 previous similar messages [ 1466.121775] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1466.139869] Lustre: Skipped 1 previous similar message [ 1467.816970] Lustre: server umount lustre-MDT0000 complete [ 1470.890424] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1470.954601] LustreError: 50326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1470.992632] LustreError: 50326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1477.100198] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.122969] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1477.163869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.367342] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.670079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.747753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1482.791821] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1483.749043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1485.623559] Lustre: DEBUG MARKER: Wait for df (36656) ... [ 1487.141690] Lustre: DEBUG MARKER: done [ 1492.963404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.983595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.999795] Lustre: Skipped 2 previous similar messages [ 1493.012420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1494.018583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.136076] Lustre: server umount lustre-OST0000 complete [ 1504.226029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1504.234380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.265218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.267740] Lustre: Skipped 2 previous similar messages [ 1508.475948] Lustre: server umount lustre-MDT0000 complete [ 1511.405773] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.441166] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1512.291620] LustreError: 50307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587649 with bad export cookie 889957285761930637 [ 1512.297139] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.301171] LustreError: 50307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1512.615333] Lustre: server umount lustre-MDT0001 complete [ 1521.630841] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1524.091845] Key type lgssc unregistered [ 1524.390949] LNet: 55244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.400097] LNetError: 55244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.417377] LNet: Removed LNI 192.168.203.150@tcp [ 1525.302914] Key type .llcrypt unregistered [ 1525.306199] Key type ._llcrypt unregistered [ 1560.439801] Key type ._llcrypt registered [ 1560.442256] Key type .llcrypt registered [ 1560.590633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1561.666146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.899299] alg: No test for adler32 (adler32-zlib) [ 1563.054106] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1563.279603] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1565.000750] Key type lgssc registered [ 1566.076700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1573.990907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1574.003785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.187533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1575.201427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1575.331732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.399651] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1588.893422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.116246] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1589.193522] Lustre: 57425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1589.675852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1594.991854] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1602.651582] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1608.175091] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1615.029029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.136794] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1615.358484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.670412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1621.802819] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1628.587800] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1634.361824] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-05 1h56m09s [ 1636.075277] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1642.908760] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1645.138364] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1645.300370] LustreError: 57435:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1665.763694] Lustre: lustre-MDT0000: Client 46085385-f6ab-4c64-950a-38950b831daf (at 192.168.203.50@tcp) reconnecting [ 1685.392108] LustreError: 57435:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1687.773435] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1689.875206] Lustre: DEBUG MARKER: done [ 1698.274157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.280156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.298159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1698.788157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.795174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.809832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.777596] Lustre: server umount lustre-OST0000 complete [ 1703.906962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.916319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.919646] Lustre: Skipped 1 previous similar message [ 1703.942035] Lustre: Skipped 1 previous similar message [ 1709.989518] Lustre: server umount lustre-MDT0000 complete [ 1713.169545] LustreError: 59289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587850 with bad export cookie 2349124242922235124 [ 1713.172259] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.186930] LustreError: 59289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1713.507054] Lustre: server umount lustre-MDT0001 complete [ 1724.719522] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1727.546984] Key type lgssc unregistered [ 1727.894073] LNet: 61757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.901676] LNetError: 61757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.919644] LNet: Removed LNI 192.168.203.150@tcp [ 1728.805631] Key type .llcrypt unregistered [ 1728.807392] Key type ._llcrypt unregistered [ 1768.892687] Key type ._llcrypt registered [ 1768.898351] Key type .llcrypt registered [ 1769.001163] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1770.314139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.468442] alg: No test for adler32 (adler32-zlib) [ 1771.689871] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1771.952678] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1773.616175] Key type lgssc registered [ 1775.036918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.992792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.021490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.247689] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.267454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.417029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.379408] Lustre: Failing over lustre-MDT0000 [ 1781.611801] Lustre: server umount lustre-MDT0000 complete [ 1788.218769] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:58:43 (1788587923) [ 1795.497295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.005259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.597258] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1807.418491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.516395] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1807.562497] Lustre: 64550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1807.890193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1811.785931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1818.116874] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1822.757271] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1828.377180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1828.541297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1828.760517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.133093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1834.577716] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1841.032636] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1845.798516] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1853.922043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1853.931543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.950169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.947690] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.948520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.981898] Lustre: Skipped 1 previous similar message [ 1854.987158] Lustre: Skipped 1 previous similar message [ 1858.688495] Lustre: server umount lustre-MDT0000 complete [ 1862.231795] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587999 with bad export cookie 7613382490106854748 [ 1862.235592] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.244195] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1862.563796] Lustre: server umount lustre-MDT0001 complete [ 1865.216154] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1868.979546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1876.690734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.312202] Lustre: 63474:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588002/real 1788588002] req@00000000237afe15 x1875470355306752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1881.341472] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.592214] Lustre: 63475:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588004/real 1788588004] req@00000000b05e22c2 x1875470355306880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1885.728392] Lustre: 63475:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588007/real 1788588007] req@000000007605e223 x1875470355307008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1887.521138] LustreError: 63472:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000949da5d5 x1875470355307776/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1887.868236] LustreError: 66398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.890296] LustreError: 66398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1887.980451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.051090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1890.981212] LustreError: 66399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.011517] LustreError: 68317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.029101] LustreError: 68317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1892.140335] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1893.025306] Lustre: 63473:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588014/real 1788588014] req@00000000d1c8ba93 x1875470355307392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1893.056748] Lustre: 63473:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1897.442913] LustreError: 68314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.470529] LustreError: 68314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1899.506074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.799211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.520986] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1905.131666] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1909.620851] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1910.254034] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1914.257657] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1920.484086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.487325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.491304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1920.491313] Lustre: Skipped 1 previous similar message [ 1920.530353] Lustre: Skipped 2 previous similar messages [ 1922.147182] Lustre: server umount lustre-MDT0000 complete [ 1925.603349] LustreError: 69055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.617133] LustreError: 69055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1925.650836] LustreError: 64540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588063 with bad export cookie 7613382490106857023 [ 1925.656684] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.667181] LustreError: 64540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1926.037045] Lustre: server umount lustre-MDT0001 complete [ 1931.586451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1939.971334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.104292] Lustre: 63476:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588068/real 1788588068] req@00000000fc1ed01b x1875470355331072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1947.138801] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.268686] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x69a82b098dff8ba2 [ 1951.278552] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 1951.589516] LustreError: 66398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.715648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.775968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1955.643658] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1956.823943] Lustre: 63474:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788588078/real 1788588078] req@000000009219ca2c x1875470355331456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788588094 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1956.874372] Lustre: 63474:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1964.108890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.529350] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1969.652668] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1975.864515] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1976.352564] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.265772] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.019398] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.025789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.038312] Lustre: Skipped 2 previous similar messages [ 1986.045727] Lustre: Skipped 1 previous similar message [ 1991.139142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.150795] Lustre: Skipped 4 previous similar messages [ 1992.321777] Lustre: server umount lustre-MDT0000 complete [ 1996.265693] LustreError: 71748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.303077] LustreError: 71748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 1996.363229] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588133 with bad export cookie 7613382490106858402 [ 1996.371567] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.378701] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1996.631602] Lustre: server umount lustre-MDT0001 complete [ 2002.679362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2005.326682] Lustre: DEBUG MARKER: files compared the same [ 2017.403740] Lustre: server umount lustre-OST0000 complete [ 2032.979837] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2035.477590] Key type lgssc unregistered [ 2035.796115] LNet: 74288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.801665] LNetError: 74288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.819935] LNet: Removed LNI 192.168.203.150@tcp [ 2036.655745] Key type .llcrypt unregistered [ 2036.657820] Key type ._llcrypt unregistered [ 2058.641499] Key type ._llcrypt registered [ 2058.644443] Key type .llcrypt registered [ 2058.768072] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2059.848943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2059.935821] alg: No test for adler32 (adler32-zlib) [ 2061.091937] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2061.276743] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2063.000225] Key type lgssc registered [ 2064.145893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.158982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2070.173728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.538543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.037851] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 2080.743169] LustreError: 75699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.775477] LustreError: 75699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2082.238161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.535497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2086.461210] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 2092.744689] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2097.459654] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2103.478986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2103.702813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.144841] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 2116.007851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2116.206854] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2124.272758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.285096] Lustre: Skipped 1 previous similar message [ 2124.287802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.376904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.387328] Lustre: Skipped 2 previous similar messages [ 2129.477249] Lustre: server umount lustre-OST0000 complete [ 2133.985194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.997355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.008738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.744713] Lustre: server umount lustre-MDT0000 complete [ 2139.622455] LustreError: 75695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.649208] LustreError: 75695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2142.316323] LustreError: 75680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588279 with bad export cookie 17065064093394746837 [ 2142.327277] LustreError: 75680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2142.328284] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.647496] Lustre: server umount lustre-MDT0001 complete [ 2152.699234] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2155.142918] Key type lgssc unregistered [ 2155.369306] LNet: 79228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.375596] LNetError: 79228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.387151] LNet: Removed LNI 192.168.203.150@tcp [ 2156.217501] Key type .llcrypt unregistered [ 2156.220112] Key type ._llcrypt unregistered [ 2181.330356] Key type ._llcrypt registered [ 2181.339429] Key type .llcrypt registered [ 2181.550322] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2183.117836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.197561] alg: No test for adler32 (adler32-zlib) [ 2184.255425] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2184.487951] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2186.176288] Key type lgssc registered [ 2187.371675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2195.191147] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2195.201191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2210.785830] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2215.904334] LustreError: 80615:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 2242.592740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.352631] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2257.384366] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.976648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.226296] Lustre: *** cfs_fail_loc=706, val=0*** [ 2265.776529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.894510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2271.215910] LustreError: 80641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.234209] LustreError: 80641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2271.322778] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2276.325472] LustreError: 80640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.353557] LustreError: 81700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.368432] LustreError: 81700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2279.865290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.346938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.553555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2293.870684] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2300.614759] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.027074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.032897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.043888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2311.138484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2311.147149] Lustre: Skipped 2 previous similar messages [ 2316.269671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.274027] Lustre: Skipped 1 previous similar message [ 2317.280291] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2317.454034] Lustre: server umount lustre-OST0000 complete [ 2321.379037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.381289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2321.384427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.392285] Lustre: Skipped 2 previous similar messages [ 2326.499086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.517029] Lustre: Skipped 2 previous similar messages [ 2335.200217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2335.600516] Lustre: server umount lustre-MDT0000 complete [ 2336.737902] LustreError: 81706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.770632] LustreError: 81706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2339.719450] LustreError: 80622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588477 with bad export cookie 5794864271242880239 [ 2339.725750] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.730571] LustreError: 80622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2339.988447] Lustre: server umount lustre-MDT0001 complete [ 2349.790667] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2352.238288] Key type lgssc unregistered [ 2352.559797] LNet: 84233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.569734] LNetError: 84233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.607970] LNet: Removed LNI 192.168.203.150@tcp [ 2353.775698] Key type .llcrypt unregistered [ 2353.779313] Key type ._llcrypt unregistered [ 2379.449629] Key type ._llcrypt registered [ 2379.452057] Key type .llcrypt registered [ 2379.549451] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2380.930090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.957768] alg: No test for adler32 (adler32-zlib) [ 2382.070694] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2382.326508] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2384.048883] Key type lgssc registered [ 2385.633200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2392.780804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2392.806935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.077458] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2405.061729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.487553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.123139] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2410.980237] LustreError: 86336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.003723] LustreError: 86336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2415.077594] LustreError: 86339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.093173] LustreError: 86339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2418.125540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2418.604160] LustreError: 86340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.644860] LustreError: 86340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2418.663707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.800429] LustreError: 86339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.835121] LustreError: 86339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2425.046785] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2428.897815] LustreError: 86335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.955918] LustreError: 86335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2435.136318] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2435.146795] Lustre: MGS: Client 9530a075-a245-401c-84df-c94c7662c850 (at 0@lo) reconnecting [ 2435.173373] LustreError: 86350:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2435.583980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.699388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2439.870809] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2456.043833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.046316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2456.077243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.084254] Lustre: Skipped 1 previous similar message [ 2461.155415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.160062] Lustre: Skipped 1 previous similar message [ 2461.560148] Lustre: server umount lustre-OST0000 complete [ 2466.274957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.276103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2466.282418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.300710] Lustre: Skipped 2 previous similar messages [ 2471.399204] LustreError: 86335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.419896] LustreError: 86335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2471.435602] Lustre: server umount lustre-MDT0000 complete [ 2475.140925] Lustre: server umount lustre-MDT0001 complete [ 2478.676952] Lustre: server umount MGS complete [ 2490.295554] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:10:25 (1788588625) [ 2521.531732] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2530.910958] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2566.932946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.865346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2581.242280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.486683] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2581.532198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2581.625694] Lustre: lustre-MDT0001: new disk, initializing [ 2581.734914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2581.760076] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2581.771554] Lustre: 93523:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2581.783361] LustreError: 93523:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2587.352985] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2600.115087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.311989] Lustre: lustre-OST0000: new disk, initializing [ 2600.315925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2600.324213] Lustre: 94291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2600.385957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.664094] Lustre: 94580:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003d04b070 x1875470994939776/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788588761 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2608.063628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2617.926595] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2624.375468] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.387854] Lustre: MGS: Client 334b8e9c-ea44-495e-8f2d-440369d6c3fb (at 0@lo) reconnecting [ 2624.402313] LustreError: 93509:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2624.445489] Lustre: 92792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2624.508796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2624.578564] Lustre: lustre-MDT0000: new disk, initializing [ 2624.704694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.743996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2624.762282] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2625.833777] Lustre: 95361:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009f8f23e2 x1875470994946816/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788588779 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2629.373457] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2631.015903] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2634.986933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2635.000511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2635.041056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2659.811963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.826717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.830880] Lustre: Skipped 1 previous similar message [ 2659.848742] Lustre: Skipped 2 previous similar messages [ 2664.197847] Lustre: server umount lustre-OST0000 complete [ 2670.049659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2670.055569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.062910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.089696] Lustre: Skipped 1 previous similar message [ 2674.503994] Lustre: server umount lustre-MDT0000 complete [ 2675.178305] LustreError: 93498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.216368] LustreError: 93498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2684.673744] Lustre: server umount lustre-MDT0001 complete [ 2688.711502] Lustre: server umount MGS complete [ 2699.858972] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:13:54 (1788588834) [ 2718.201894] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2721.422555] Key type lgssc unregistered [ 2721.807762] LNet: 97860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.822958] LNetError: 97860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.840056] LNet: Removed LNI 192.168.203.150@tcp [ 2722.885587] Key type .llcrypt unregistered [ 2722.888417] Key type ._llcrypt unregistered [ 2740.040703] Key type ._llcrypt registered [ 2740.048046] Key type .llcrypt registered [ 2740.180034] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2741.832854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2741.948707] alg: No test for adler32 (adler32-zlib) [ 2743.257765] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2743.593975] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2745.352621] Key type lgssc registered [ 2747.522593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2757.343804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2757.377676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.483300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.514069] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2768.867838] LustreError: 99080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.877921] LustreError: 99080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2772.791487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.010690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.546424] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2786.750436] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2792.722444] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2799.875210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.446073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.547114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2807.849673] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2814.714214] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2828.257966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.272728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.277238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.287780] Lustre: Skipped 1 previous similar message [ 2833.380032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2833.384691] Lustre: Skipped 1 previous similar message [ 2834.118844] Lustre: server umount lustre-OST0000 complete [ 2838.497588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.501450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2838.532914] Lustre: Skipped 1 previous similar message [ 2843.617158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.623556] Lustre: Skipped 2 previous similar messages [ 2844.584347] Lustre: server umount lustre-MDT0000 complete [ 2848.274284] LustreError: 99059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588985 with bad export cookie 3831397434379322172 [ 2848.286105] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2848.304880] LustreError: 99059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2848.683214] Lustre: server umount lustre-MDT0001 complete [ 2859.083170] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2861.845351] Key type lgssc unregistered [ 2862.172468] LNet: 102632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.197308] LNetError: 102632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.217067] LNet: Removed LNI 192.168.203.150@tcp [ 2863.026374] Key type .llcrypt unregistered [ 2863.030108] Key type ._llcrypt unregistered [ 2876.609315] Key type ._llcrypt registered [ 2876.611071] Key type .llcrypt registered [ 2876.679744] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2877.692735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2877.720485] alg: No test for adler32 (adler32-zlib) [ 2878.768380] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2879.070487] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2880.720163] Key type lgssc registered [ 2882.131045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2888.176298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.178109] LustreError: 103691:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2888.185178] Lustre: Skipped 1 previous similar message [ 2888.185199] LustreError: 103689:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2888.204723] LustreError: 103691:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2888.210039] LustreError: 103691:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2888.216833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.219477] LustreError: 103691:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2889.738842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.397468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.730681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.567657] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2909.914994] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing lsmod [ 2912.646941] LustreError: 104898:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2912.652782] LustreError: 104899:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2912.657136] LustreError: 104898:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2912.663534] LustreError: 104898:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2912.670118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2912.674505] LustreError: 104898:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2913.147617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.371938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.373352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.407084] Lustre: Skipped 1 previous similar message [ 2918.431110] Lustre: Skipped 2 previous similar messages [ 2921.959139] Lustre: server umount lustre-MDT0000 complete [ 2923.498510] LustreError: 104164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.515130] LustreError: 104164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2925.607423] LustreError: 103730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.628478] LustreError: 103730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2925.768124] LustreError: 103708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589063 with bad export cookie 6035805241103205819 [ 2925.780958] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2925.790488] LustreError: 103708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2927.712315] LustreError: 104161:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f8754f99 x1875471516061696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2927.921467] Lustre: server umount lustre-MDT0001 complete [ 2940.923889] Lustre: server umount lustre-OST0000 complete [ 2948.738360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.110234] LustreError: 105963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.172565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.614421] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2954.214688] LustreError: 105964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.329796] LustreError: 105963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.666866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.083672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2967.419167] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2974.945671] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2980.790526] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2986.851358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.152485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.871349] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2994.359431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2999.342767] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3017.188358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3017.192130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.205516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.210982] Lustre: Skipped 2 previous similar messages [ 3022.307997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.321267] Lustre: Skipped 2 previous similar messages [ 3022.993672] Lustre: server umount lustre-OST0000 complete [ 3027.430194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.431742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3027.441593] Lustre: Skipped 1 previous similar message [ 3032.547697] LustreError: 106951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.742771] Lustre: server umount lustre-MDT0000 complete [ 3036.245030] LustreError: 105944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589173 with bad export cookie 6035805241103206519 [ 3036.247580] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3036.264929] LustreError: 105944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3036.776950] Lustre: server umount lustre-MDT0001 complete [ 3047.100957] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3049.827483] Key type lgssc unregistered [ 3050.124749] LNet: 109520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3050.140265] LNetError: 109520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3050.161170] LNet: Removed LNI 192.168.203.150@tcp [ 3051.094186] Key type .llcrypt unregistered [ 3051.108215] Key type ._llcrypt unregistered [ 3075.311285] Key type ._llcrypt registered [ 3075.313265] Key type .llcrypt registered [ 3075.416202] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3076.738412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3076.791788] alg: No test for adler32 (adler32-zlib) [ 3077.843332] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3078.173603] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3079.873551] Key type lgssc registered [ 3081.046666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3088.156507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3088.181073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.597988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.953927] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3099.112557] LustreError: 110935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.144663] LustreError: 110935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3104.056211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.225898] LustreError: 110934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.451532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3108.785868] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3117.707835] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3123.305626] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.385973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.781391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.794376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3141.962701] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3149.889372] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.800429] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3159.413406] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3164.643813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.649664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3164.675727] Lustre: Skipped 1 previous similar message [ 3169.771367] LustreError: 112757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.797485] LustreError: 112757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3169.909226] Lustre: server umount lustre-OST0000 complete [ 3174.897357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.902519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.929556] Lustre: Skipped 1 previous similar message [ 3174.932170] Lustre: Skipped 1 previous similar message [ 3179.972089] Lustre: server umount lustre-MDT0000 complete [ 3180.005334] LustreError: 112132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.031271] LustreError: 112132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3183.546958] LustreError: 113467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589320 with bad export cookie 10603318449260274447 [ 3183.557837] LustreError: 113467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3183.560934] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3183.912040] Lustre: server umount lustre-MDT0001 complete [ 3195.810058] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3198.630780] Key type lgssc unregistered [ 3198.940189] LNet: 114572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3198.957045] LNetError: 114572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3198.980249] LNet: Removed LNI 192.168.203.150@tcp [ 3199.818949] Key type .llcrypt unregistered [ 3199.821183] Key type ._llcrypt unregistered [ 3215.484875] Key type ._llcrypt registered [ 3215.486375] Key type .llcrypt registered [ 3215.615489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3216.887728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3216.906552] alg: No test for adler32 (adler32-zlib) [ 3217.924788] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3218.112788] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3219.768148] Key type lgssc registered [ 3220.698837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.173058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.213945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.568048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.918430] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3237.858644] LustreError: 115787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.882863] LustreError: 115787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3240.360672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.798771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.012444] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3250.951567] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.133337] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3264.097334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.568668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.882527] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3272.780503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3278.763609] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.131849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.143978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.172463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.164916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.178631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.182380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.362826] Lustre: server umount lustre-OST0000 complete [ 3298.275540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.279667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3298.316689] Lustre: Skipped 1 previous similar message [ 3298.352073] Lustre: Skipped 2 previous similar messages [ 3303.394272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.274944] Lustre: server umount lustre-MDT0000 complete [ 3308.518594] LustreError: 115783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.562303] LustreError: 115783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3308.646361] LustreError: 115769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589446 with bad export cookie 11475999202905316911 [ 3308.655647] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.671874] LustreError: 115769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3309.225931] Lustre: server umount lustre-MDT0001 complete [ 3321.132175] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3324.125760] Key type lgssc unregistered [ 3324.381694] LNet: 119322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3324.389712] LNetError: 119322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3324.404786] LNet: Removed LNI 192.168.203.150@tcp [ 3325.304303] Key type .llcrypt unregistered [ 3325.307309] Key type ._llcrypt unregistered [ 3348.621726] Key type ._llcrypt registered [ 3348.626437] Key type .llcrypt registered [ 3348.778969] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3349.691404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.826795] alg: No test for adler32 (adler32-zlib) [ 3350.839788] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3351.058115] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3352.728188] Key type lgssc registered [ 3354.164601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3361.923391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3361.942601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.348416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.639705] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3373.545054] LustreError: 120737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.565328] LustreError: 120737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3375.343655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.821865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.312734] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3388.735192] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3394.334862] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3400.844677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.258985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.961653] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3408.459732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3415.318630] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3425.778736] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3429.622116] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3429.627401] Lustre: Skipped 2 previous similar messages [ 3431.007763] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3431.043540] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3434.398465] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3434.406983] Lustre: Skipped 2 previous similar messages [ 3438.469667] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3438.478230] Lustre: Skipped 1 previous similar message [ 3445.413462] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3445.430170] Lustre: Skipped 2 previous similar messages [ 3446.627881] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp [ 3456.328300] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.50/24@tcp in log lustre-MDT0000 [ 3456.334822] Lustre: Skipped 2 previous similar messages [ 3466.304435] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.50/24@tcp [ 3466.320244] Lustre: Skipped 1 previous similar message [ 3474.527493] Lustre: server umount lustre-OST0000 complete [ 3477.987052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3477.994380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.010625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.035064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.036086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.160801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.170015] Lustre: Skipped 1 previous similar message [ 3490.279120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.287669] Lustre: Skipped 1 previous similar message [ 3491.296393] Lustre: 120048:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589612/real 1788589612] req@0000000043a3c8d3 x1875472011038208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788589628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3491.336848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.323868] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3492.602274] Lustre: server umount lustre-MDT0000 complete [ 3493.408091] Lustre: 120048:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589614/real 1788589614] req@00000000b487a2d0 x1875472011038720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788589630 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3495.737518] LustreError: 120720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589633 with bad export cookie 2932410753684980880 [ 3495.741194] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.748698] LustreError: 120720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3496.039711] Lustre: server umount lustre-MDT0001 complete [ 3505.809239] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3508.603502] Key type lgssc unregistered [ 3508.889930] LNet: 125633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3508.901414] LNetError: 125633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3509.940762] LNet: Removed LNI 192.168.203.150@tcp [ 3510.827454] Key type .llcrypt unregistered [ 3510.831652] Key type ._llcrypt unregistered [ 3539.635461] Key type ._llcrypt registered [ 3539.648332] Key type .llcrypt registered [ 3539.853607] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3541.188963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3541.221514] alg: No test for adler32 (adler32-zlib) [ 3542.586832] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3542.932548] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3544.672206] Key type lgssc registered [ 3545.942613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3554.437375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3554.453368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.799692] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3555.821350] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.50/24@tcp [ 3555.851857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.119456] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3566.059634] LustreError: 127190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.083071] LustreError: 127190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3570.647815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.995057] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3571.031934] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.50/24@tcp [ 3571.075752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.311757] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3583.971560] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3589.749996] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3597.609343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.958388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.014521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3604.592079] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3613.337541] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3632.108921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.110701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.122196] Lustre: Skipped 1 previous similar message [ 3632.133764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3633.671574] Lustre: server umount lustre-OST0000 complete [ 3642.337520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.341137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3642.368957] Lustre: Skipped 1 previous similar message [ 3642.373014] Lustre: Skipped 1 previous similar message [ 3644.228762] Lustre: server umount lustre-MDT0000 complete [ 3647.462806] LustreError: 127924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.489740] LustreError: 127924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3648.028528] LustreError: 127169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589785 with bad export cookie 561646103595574155 [ 3648.042330] LustreError: 127169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3648.049420] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.579495] Lustre: server umount lustre-MDT0001 complete [ 3660.139646] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3663.194655] Key type lgssc unregistered [ 3663.550049] LNet: 130781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3663.563692] LNetError: 130781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.581705] LNet: Removed LNI 192.168.203.150@tcp [ 3664.718126] Key type .llcrypt unregistered [ 3664.723190] Key type ._llcrypt unregistered [ 3693.045069] Key type ._llcrypt registered [ 3693.050249] Key type .llcrypt registered [ 3693.195785] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3709.932895] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3711.186377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3711.368761] alg: No test for adler32 (adler32-zlib) [ 3712.652769] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3712.950424] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3714.696216] Key type lgssc registered [ 3716.027176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3753.515781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3753.549855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.833948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3754.877422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3754.967414] Lustre: lustre-MDT0000: new disk, initializing [ 3755.072474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.091431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3760.010354] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3771.254771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.418866] Lustre: 134183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3771.499398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3771.505218] Lustre: Skipped 1 previous similar message [ 3771.652640] Lustre: lustre-MDT0001: new disk, initializing [ 3771.734310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3771.784125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3771.803778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3777.108202] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3786.962109] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.607202] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3799.227662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.587415] Lustre: lustre-OST0000: new disk, initializing [ 3799.601155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3799.615233] Lustre: 136152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3799.720473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.276492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3805.299477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3805.416790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3805.926346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3815.338431] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.604786] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3819.905788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3824.841609] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3825.059395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3831.271504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3831.282209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.297594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.304555] Lustre: Skipped 1 previous similar message [ 3833.185353] Lustre: server umount lustre-OST0000 complete [ 3838.435663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3838.443536] LustreError: Skipped 1 previous similar message [ 3838.445549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.460290] Lustre: Skipped 1 previous similar message [ 3838.476399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.513111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.517038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.932161] Lustre: server umount lustre-MDT0000 complete [ 3847.817422] LustreError: 134173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589985 with bad export cookie 17820705257376560319 [ 3847.819906] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3847.834306] LustreError: 134173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.302036] Lustre: server umount lustre-MDT0001 complete [ 3857.979621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.717129] LustreError: 138368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.746514] LustreError: 138368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3858.905037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.051499] LustreError: 138369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.046310] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3869.178968] LustreError: 138368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.277672] LustreError: 138368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.911672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.496637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3881.700582] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3889.340493] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3896.333959] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3903.997719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.642170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.718361] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3923.115913] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3929.973414] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3930.299876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3935.679841] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3935.935924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3945.129768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3945.315479] Lustre: lustre-OST0001: new disk, initializing [ 3945.321616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3945.327692] Lustre: 141743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3945.481620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.620259] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3953.748169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3953.762726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3953.882896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3961.041375] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3961.288156] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3966.488378] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3966.726842] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3972.685692] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9becc8ea3800.ost_server_uuid 50 [ 3974.499700] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9becc8ea3800.ost_server_uuid in IDLE state after 0 sec [ 3984.874170] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.879251] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.894212] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3984.896859] LustreError: Skipped 1 previous similar message [ 3984.917959] Lustre: Skipped 2 previous similar messages [ 3990.002468] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3990.012435] Lustre: Skipped 1 previous similar message [ 3990.390302] Lustre: server umount lustre-OST0001 complete [ 3995.122184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.128662] LustreError: 140191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.153833] Lustre: Skipped 2 previous similar messages [ 3995.183707] LustreError: 140191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4000.230748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.254184] Lustre: Skipped 3 previous similar messages [ 4000.679661] Lustre: server umount lustre-OST0000 complete [ 4005.346626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.365043] Lustre: Skipped 1 previous similar message [ 4010.735532] Lustre: server umount lustre-MDT0000 complete [ 4015.061435] LustreError: 138349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590152 with bad export cookie 17820705257376561201 [ 4015.063231] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.078139] LustreError: 138349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4015.437709] Lustre: server umount lustre-MDT0001 complete [ 4036.189781] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4039.610192] Key type lgssc unregistered [ 4039.975529] LNet: 144742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4039.983917] LNetError: 144742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4039.997391] LNet: Removed LNI 192.168.203.150@tcp [ 4040.929460] Key type .llcrypt unregistered [ 4040.931878] Key type ._llcrypt unregistered [ 4081.582921] Key type ._llcrypt registered [ 4081.587527] Key type .llcrypt registered [ 4081.724775] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4097.677983] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4099.251904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4099.263259] alg: No test for adler32 (adler32-zlib) [ 4100.355633] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4100.619679] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4102.304177] Key type lgssc registered [ 4103.668812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4136.734628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4136.766345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.040802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4138.064216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4138.141757] Lustre: lustre-MDT0000: new disk, initializing [ 4138.230805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.244423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4142.954239] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4153.862882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.969863] Lustre: 148896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4154.030868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4154.043869] Lustre: Skipped 1 previous similar message [ 4154.120574] Lustre: lustre-MDT0001: new disk, initializing [ 4154.244440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4154.298565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4154.320335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4158.642111] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4168.103690] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4173.899768] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4180.634718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4180.963393] Lustre: lustre-OST0000: new disk, initializing [ 4180.976346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4180.991918] Lustre: 150864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4181.154524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.228439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.239854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4187.384785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4187.443489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4196.776894] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4201.813687] Lustre: Failing over lustre-OST0000 [ 4201.967721] Lustre: server umount lustre-OST0000 complete [ 4219.360112] Lustre: 147121:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590340/real 1788590340] req@00000000dce077f6 x1875472797434368/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788590356 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4219.360728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.409442] Lustre: 147121:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4219.469831] Lustre: Skipped 1 previous similar message [ 4220.387345] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590341/real 1788590341] req@00000000f8d51246 x1875472797435392/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788590357 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4220.400524] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4221.163791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.492994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.504227] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4224.528157] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590345/real 1788590345] req@000000007f24c67c x1875472797436544/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788590361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4229.091827] Lustre: 147121:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590350/real 1788590350] req@00000000a556aebd x1875472797437568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788590366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4229.141372] Lustre: 147121:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4229.471824] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4229.612467] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4230.376023] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4230.376597] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4234.223548] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590355/real 1788590355] req@000000001ee1f98a x1875472797438848/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788590371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4234.256652] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4234.337374] Lustre: Failing over lustre-MDT0000 [ 4234.729297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.744911] Lustre: Skipped 2 previous similar messages [ 4234.877646] Lustre: server umount lustre-MDT0000 complete [ 4239.841601] LustreError: 148906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.861898] LustreError: 148906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4240.597150] LustreError: 150872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.964830] LustreError: 148901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.993350] LustreError: 148901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4250.086466] LustreError: 148901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.105787] LustreError: 148901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4254.087026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.272251] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4254.563674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.646567] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4256.020872] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4259.354932] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4259.832413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4259.837518] Lustre: Skipped 1 previous similar message [ 4259.859898] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4259.929350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4270.063698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.091266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4270.098915] Lustre: Skipped 1 previous similar message [ 4272.895556] Lustre: server umount lustre-OST0000 complete [ 4280.290574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4280.294447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.295996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4280.324812] Lustre: Skipped 2 previous similar messages [ 4283.021827] Lustre: server umount lustre-MDT0000 complete [ 4285.410504] LustreError: 150872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.432687] LustreError: 150872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4286.126922] LustreError: 148888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590423 with bad export cookie 2648229283838100299 [ 4286.128320] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4286.139831] LustreError: 148888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4286.433400] Lustre: server umount lustre-MDT0001 complete [ 4296.169136] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4298.994329] Key type lgssc unregistered [ 4299.336530] LNet: 154515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4299.344145] LNetError: 154515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4299.366894] LNet: Removed LNI 192.168.203.150@tcp [ 4300.356952] Key type .llcrypt unregistered [ 4300.360778] Key type ._llcrypt unregistered [ 4335.352742] Key type ._llcrypt registered [ 4335.359408] Key type .llcrypt registered [ 4335.482886] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4336.734419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4336.771187] alg: No test for adler32 (adler32-zlib) [ 4337.939236] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4338.180825] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4339.816681] Key type lgssc registered [ 4341.030554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4348.467429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4348.503494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.114812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.967865] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4359.715813] LustreError: 156032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.728953] LustreError: 156032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4363.364218] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4364.776067] LustreError: 156031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.034707] LustreError: 156060:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4368.051123] LustreError: 156060:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4368.064904] LustreError: 156060:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4368.615709] Lustre: server umount lustre-MDT0000 complete [ 4387.049375] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4396.128391] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4429.209603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.414290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4429.443791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4429.504477] Lustre: lustre-MDT0000: new disk, initializing [ 4429.563602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.596696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4433.374996] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4442.581182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.726948] Lustre: 159362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4442.793725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4442.797821] Lustre: Skipped 1 previous similar message [ 4442.870351] Lustre: lustre-MDT0001: new disk, initializing [ 4442.982243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4443.039412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4443.052857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4447.184787] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4455.859131] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4460.951810] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4466.940953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.165225] Lustre: lustre-OST0000: new disk, initializing [ 4467.167757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4467.172905] Lustre: 161334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4467.237369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.949580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4472.981223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4473.184656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4473.612632] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4482.518612] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4487.182588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4488.981661] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:43:44 (1788590624) [ 4719.073148] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.073516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4719.087462] Lustre: Skipped 1 previous similar message [ 4719.106569] Lustre: Skipped 2 previous similar messages [ 4721.836904] Lustre: server umount lustre-MDT0000 complete [ 4724.192903] LustreError: 160647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.235839] LustreError: 160647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4725.351047] LustreError: 160168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590862 with bad export cookie 8206540971554783290 [ 4725.357316] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4725.361095] LustreError: 160168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4729.313951] LustreError: 159372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.314316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.345378] Lustre: Skipped 1 previous similar message [ 4729.349596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4731.620938] Lustre: server umount lustre-MDT0001 complete [ 4749.280164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4749.505161] Lustre: server umount lustre-OST0000 complete [ 4758.961820] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4767.372186] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4799.554616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.736921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4799.756531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4799.820997] Lustre: lustre-MDT0000: new disk, initializing [ 4799.891762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.921788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4803.776946] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4812.528182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.653290] Lustre: 165988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4812.711855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4812.717903] Lustre: Skipped 1 previous similar message [ 4812.780771] Lustre: lustre-MDT0001: new disk, initializing [ 4812.854352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4812.881578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4812.894995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4816.477766] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4825.122877] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4830.422531] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4836.248681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4836.513300] Lustre: lustre-OST0000: new disk, initializing [ 4836.516549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4836.521858] Lustre: 167954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4836.588594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.228894] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4844.618840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4844.632932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4844.723827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4851.237199] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4855.624699] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4855.797794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4860.220449] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4860.505575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4865.509528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.511259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4865.537853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4865.548268] Lustre: Skipped 2 previous similar messages [ 4868.489046] Lustre: server umount lustre-OST0000 complete [ 4874.209462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4874.221583] LustreError: Skipped 1 previous similar message [ 4874.225733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.244946] Lustre: Skipped 1 previous similar message [ 4874.249637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4880.473637] Lustre: server umount lustre-MDT0000 complete [ 4880.870927] LustreError: 166002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.908419] LustreError: 166002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4883.767207] LustreError: 165983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591021 with bad export cookie 8206540971554942120 [ 4883.773539] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.778506] LustreError: 165983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4884.036634] Lustre: server umount lustre-MDT0001 complete [ 4891.668749] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:50:27 (1788591027) [ 4905.915136] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4914.520120] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4941.578532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.734699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4941.753969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4941.813617] Lustre: lustre-MDT0000: new disk, initializing [ 4941.875229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.896383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4945.415521] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4954.000676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.099887] Lustre: 172719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4954.345965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4958.351840] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4967.116759] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4972.424986] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4978.207413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4978.598055] Lustre: lustre-OST0000: new disk, initializing [ 4978.603884] Lustre: Skipped 1 previous similar message [ 4978.617634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4978.628382] Lustre: Skipped 2 previous similar messages [ 4978.640975] Lustre: 174697:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4985.057170] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4987.437092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4987.444475] Lustre: Skipped 1 previous similar message [ 4987.447541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4987.513184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4993.989355] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5002.721563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.746312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.766695] Lustre: Skipped 1 previous similar message [ 5002.781649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.789165] Lustre: Skipped 3 previous similar messages [ 5008.088921] Lustre: server umount lustre-OST0000 complete [ 5012.966643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.978790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5013.001146] Lustre: Skipped 2 previous similar messages [ 5013.040331] Lustre: Skipped 3 previous similar messages [ 5026.272527] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5026.576774] Lustre: server umount lustre-MDT0000 complete [ 5028.321708] LustreError: 172731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.345938] LustreError: 172731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5030.183689] LustreError: 172713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591167 with bad export cookie 8206540971554942995 [ 5030.185201] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5030.193434] LustreError: 172713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5030.567389] Lustre: server umount lustre-MDT0001 complete [ 5038.210105] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:52:53 (1788591173) [ 5053.870697] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5064.219809] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5096.909614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.260568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5097.332177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5097.523517] Lustre: lustre-MDT0000: new disk, initializing [ 5097.682145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.687222] Lustre: Skipped 2 previous similar messages [ 5097.702310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5102.576915] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5111.903362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.016633] Lustre: 179021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5112.276822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5115.970668] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5124.567287] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5129.912594] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5136.442760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.812637] Lustre: 180996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5142.765195] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5143.241043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5150.970710] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5158.470706] Lustre: server umount lustre-OST0000 complete [ 5163.490128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5163.500247] LustreError: Skipped 1 previous similar message [ 5163.508950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.540954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5163.545303] Lustre: Skipped 4 previous similar messages [ 5175.266535] Lustre: 155341:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788591296/real 1788591296] req@000000001d6ae602 x1875473046709760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788591312 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5176.288167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5176.485678] Lustre: server umount lustre-MDT0000 complete [ 5179.550795] LustreError: 179012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591316 with bad export cookie 8206540971554944087 [ 5179.555640] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5179.564044] LustreError: 179012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5179.801628] Lustre: server umount lustre-MDT0001 complete [ 5189.655849] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5192.139411] Key type lgssc unregistered [ 5192.408383] LNet: 182851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5192.413549] LNetError: 182851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5192.431342] LNet: Removed LNI 192.168.203.150@tcp [ 5193.097428] Key type .llcrypt unregistered [ 5193.099626] Key type ._llcrypt unregistered [ 5213.543171] Key type ._llcrypt registered [ 5213.545755] Key type .llcrypt registered [ 5213.647610] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5214.579717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5214.669405] alg: No test for adler32 (adler32-zlib) [ 5215.820889] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5216.213322] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5217.896154] Key type lgssc registered [ 5218.960381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5225.649323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5225.677064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.112728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.161633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5236.595089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.707274] LustreError: 184244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.730589] LustreError: 184244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5236.922606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5241.115501] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5247.734742] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5252.583932] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5258.660391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.102643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.291672] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5272.197233] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5273.464260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5293.026126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5293.043383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.071426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.051041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.057539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.088671] Lustre: Skipped 1 previous similar message [ 5296.301393] Lustre: server umount lustre-OST0000 complete [ 5303.780087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5303.785637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.801289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5306.444978] Lustre: server umount lustre-MDT0000 complete [ 5309.412933] LustreError: 184238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5309.439557] LustreError: 184238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5309.826146] LustreError: 184223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591447 with bad export cookie 13003616261337136831 [ 5309.834890] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.842779] LustreError: 184223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5310.178317] Lustre: server umount lustre-MDT0001 complete [ 5320.397620] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5323.391284] Key type lgssc unregistered [ 5323.803355] LNet: 187795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5323.811376] LNetError: 187795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5323.822976] LNet: Removed LNI 192.168.203.150@tcp [ 5324.824745] Key type .llcrypt unregistered [ 5324.827992] Key type ._llcrypt unregistered [ 5352.098198] Key type ._llcrypt registered [ 5352.099957] Key type .llcrypt registered [ 5352.217291] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5353.174915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5353.224401] alg: No test for adler32 (adler32-zlib) [ 5354.269621] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5354.469286] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5356.152250] Key type lgssc registered [ 5357.099887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5364.095889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5364.121976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.775551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.554748] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5375.981614] LustreError: 189220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.997054] LustreError: 189220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5380.237985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.652916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5384.966401] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5391.438613] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5396.545475] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5402.277587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.542993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.270778] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5410.744635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5415.359778] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5423.330932] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5426.159240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.165311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.172748] LustreError: Skipped 1 previous similar message [ 5426.174611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.219044] Lustre: Skipped 1 previous similar message [ 5428.429794] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5428.445787] Lustre: Skipped 1 previous similar message [ 5428.775571] Lustre: server umount lustre-OST0000 complete [ 5435.654652] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9bece0c99000.ost_server_uuid 50 [ 5437.212350] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5452.258546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5452.266598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.287548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5455.850093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.407472] Lustre: server umount lustre-MDT0000 complete [ 5460.763611] LustreError: 189200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591598 with bad export cookie 9293724206013216165 [ 5460.766137] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.780160] LustreError: 189200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5461.063837] Lustre: server umount lustre-MDT0001 complete [ 5468.484400] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:00:03 (1788591603) [ 5474.215287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.659161] LustreError: 193134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.729444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.054420] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5479.907639] LustreError: 193135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.030615] LustreError: 193134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.275687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.548071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.568471] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5498.389388] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5504.001451] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5509.829883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.145971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.322407] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5520.439526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5521.416100] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5528.967027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5529.249403] Lustre: lustre-OST0001: new disk, initializing [ 5529.263065] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5529.273219] Lustre: 195985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5529.378097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5536.117903] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5536.432555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5536.440559] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5536.570198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5544.954729] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5550.829758] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5550.847841] Lustre: Skipped 2 previous similar messages [ 5551.076674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5551.077752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.096680] LustreError: Skipped 1 previous similar message [ 5551.114671] Lustre: Skipped 1 previous similar message [ 5553.720172] Lustre: server umount lustre-OST0000 complete [ 5555.915756] LustreError: 196793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.625272] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5558.813531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5562.525195] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5562.715741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5566.155960] LustreError: 196352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.174833] LustreError: 196352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5577.700660] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5577.701790] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.715760] LustreError: Skipped 1 previous similar message [ 5577.729037] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5577.734526] Lustre: Skipped 5 previous similar messages [ 5581.979471] Lustre: server umount lustre-OST0001 complete [ 5587.937426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5587.940279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.957639] Lustre: Skipped 2 previous similar messages [ 5598.185269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.189847] Lustre: Skipped 5 previous similar messages [ 5599.202864] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5599.404921] Lustre: server umount lustre-MDT0000 complete [ 5602.430031] LustreError: 193116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591739 with bad export cookie 9293724206013217236 [ 5602.436606] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5602.439485] LustreError: 193116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5602.805618] Lustre: server umount lustre-MDT0001 complete [ 5623.118302] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:02:38 (1788591758) [ 5628.600178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.964320] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5628.987678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5629.162300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.785765] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5639.692926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.789145] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5639.842834] Lustre: 199423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5644.238386] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5651.949585] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5656.987819] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5663.658350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5663.869539] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5670.027927] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5674.154844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5676.870229] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5682.931026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.021072] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5683.596856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5688.720323] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5695.500543] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5699.048591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.063786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.069027] Lustre: Skipped 1 previous similar message [ 5704.425280] Lustre: server umount lustre-OST0000 complete [ 5706.449521] LustreError: 201274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.467893] LustreError: 201274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5724.647570] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5726.483458] Lustre: server umount lustre-OST0001 complete [ 5732.321507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.343043] Lustre: Skipped 3 previous similar messages [ 5737.006915] Lustre: server umount lustre-MDT0000 complete [ 5740.006286] LustreError: 199430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.036474] LustreError: 199430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5740.409391] LustreError: 199415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591877 with bad export cookie 9293724206013218643 [ 5740.412154] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5740.432312] LustreError: 199415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5740.864793] Lustre: server umount lustre-MDT0001 complete [ 5763.472526] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:04:58 (1788591898) [ 5778.546695] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5787.338382] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5817.703531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.947112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5817.987280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5818.107631] Lustre: lustre-MDT0000: new disk, initializing [ 5818.244851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.254813] Lustre: Skipped 3 previous similar messages [ 5818.280877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5823.203433] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5833.509949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.612394] Lustre: 207513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5833.670471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5833.673835] Lustre: Skipped 1 previous similar message [ 5833.737770] Lustre: lustre-MDT0001: new disk, initializing [ 5833.845770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5833.860867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5837.989397] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5846.542817] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5851.750324] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5857.585608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.987930] Lustre: lustre-OST0000: new disk, initializing [ 5857.991969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5858.003557] Lustre: 209484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5863.497188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5863.513688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5863.595783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5863.920774] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5872.279609] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5876.358762] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5876.594937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5880.819838] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5881.042989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5884.393270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5884.397547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.403236] LustreError: Skipped 3 previous similar messages [ 5884.404788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5884.404797] Lustre: Skipped 8 previous similar messages [ 5884.454526] Lustre: Skipped 2 previous similar messages [ 5897.184268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5897.331410] Lustre: server umount lustre-OST0000 complete [ 5906.882761] Lustre: server umount lustre-MDT0000 complete [ 5909.987751] LustreError: 208336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.008312] LustreError: 208336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5910.109327] LustreError: 207507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592047 with bad export cookie 9293724206013220428 [ 5910.118705] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.440359] Lustre: server umount lustre-MDT0001 complete [ 5917.429307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.957112] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5927.426799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.334414] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5937.566816] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5941.901632] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5947.492651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5947.770359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.774766] Lustre: Skipped 4 previous similar messages [ 5953.126608] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5959.804716] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5963.821541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5965.109823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5969.221595] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5969.439458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5973.476279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.486353] Lustre: server umount lustre-OST0000 complete [ 5982.580494] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5986.837633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6014.554696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.912150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.935773] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6020.834598] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6025.203608] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.228600] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6028.011781] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.768314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6040.547770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6040.561258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6040.574208] LustreError: Skipped 1 previous similar message [ 6040.577531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6040.591664] Lustre: Skipped 5 previous similar messages [ 6040.607907] Lustre: Skipped 11 previous similar messages [ 6043.825923] Lustre: server umount lustre-OST0000 complete [ 6053.946865] Lustre: server umount lustre-MDT0000 complete [ 6055.905349] LustreError: 211701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.923362] LustreError: 211701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6057.583230] LustreError: 211682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592195 with bad export cookie 9293724206013221310 [ 6057.585503] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6057.590077] LustreError: 211682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6057.859265] Lustre: server umount lustre-MDT0001 complete [ 6065.782317] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:10:01 (1788592201) [ 6072.105994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.881158] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6085.232471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.965730] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6099.125962] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6105.156874] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6112.244979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.604327] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6125.344744] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6129.299143] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6129.495607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6133.823759] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6134.044238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6139.815744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6140.063376] Lustre: lustre-OST0001: new disk, initializing [ 6140.066236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6140.070702] Lustre: 221042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6146.133552] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6146.615363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6146.623726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6146.688095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6154.567665] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6158.469670] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6158.650205] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6162.961921] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6163.199486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6179.296149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6179.379154] Lustre: server umount lustre-OST0001 complete [ 6185.122761] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6185.278669] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6188.876962] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6189.063517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6216.989039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.238281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6217.248931] Lustre: Skipped 5 previous similar messages [ 6222.320809] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6222.339996] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6222.341521] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6222.370072] Lustre: Skipped 1 previous similar message [ 6223.137631] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6230.318403] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6231.672714] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6237.671243] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6237.678496] LustreError: Skipped 1 previous similar message [ 6239.461353] Lustre: server umount lustre-OST0001 complete [ 6261.044563] Lustre: server umount lustre-OST0000 complete [ 6270.862599] Lustre: server umount lustre-MDT0000 complete [ 6274.396894] LustreError: 217654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592411 with bad export cookie 9293724206013222276 [ 6274.409940] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.412933] LustreError: 217654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6280.875549] Lustre: server umount lustre-MDT0001 complete [ 6301.295357] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:13:56 (1788592436) [ 6306.715987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.975701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6307.017966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6310.567392] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6317.825512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.965552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6318.019793] Lustre: 226282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6321.470258] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6327.070369] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6332.775585] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6339.739070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6339.991725] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6346.650253] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6354.641199] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6362.744339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.810950] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6367.394304] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6373.600662] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6375.038617] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6379.732217] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6379.963621] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6384.524886] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6384.804530] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6390.316037] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9becc3ea9000.ost_server_uuid 50 [ 6391.713437] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9becc3ea9000.ost_server_uuid in IDLE state after 0 sec [ 6392.574750] Lustre: Permanently deactivating lustre-OST0001 [ 6392.594508] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6396.140894] Lustre: Permanently reactivating lustre-OST0001 [ 6404.577908] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.585412] Lustre: Skipped 10 previous similar messages [ 6404.589256] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6404.593315] Lustre: Skipped 17 previous similar messages [ 6406.518392] Lustre: server umount lustre-OST0001 complete [ 6410.216182] LustreError: 228127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.230819] LustreError: 228127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 6416.545980] Lustre: server umount lustre-OST0000 complete [ 6426.035712] Lustre: server umount lustre-MDT0000 complete [ 6429.498399] LustreError: 226274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592566 with bad export cookie 9293724206013223564 [ 6429.508763] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6429.520649] LustreError: 226274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6429.848883] Lustre: server umount lustre-MDT0001 complete [ 6450.173340] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:16:25 (1788592585) [ 6456.160836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6456.440222] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6456.476897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6456.500491] Lustre: Skipped 5 previous similar messages [ 6460.501246] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6467.690510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.784265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6467.828740] Lustre: 232971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6471.621665] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6477.163808] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6480.792562] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6486.107349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6486.311572] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6486.330455] Lustre: Permanently deactivating lustre-OST0000 [ 6486.352445] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6492.058734] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6495.305877] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6495.401546] LustreError: 235374:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6498.289985] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6504.946484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.021277] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6509.678655] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6513.740154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6514.963955] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6516.619167] Lustre: Permanently reactivating lustre-OST0000 [ 6516.665096] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6516.670978] Lustre: Skipped 2 previous similar messages [ 6519.391375] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6519.406860] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.435749] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6519.462843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.494200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6534.122956] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6534.128666] LustreError: Skipped 8 previous similar messages [ 6538.816151] Lustre: server umount lustre-OST0001 complete [ 6548.205631] Lustre: server umount lustre-OST0000 complete [ 6557.707230] Lustre: server umount lustre-MDT0000 complete [ 6560.733622] LustreError: 232964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592698 with bad export cookie 9293724206013225216 [ 6560.734780] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6560.745892] LustreError: 232964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6560.980445] Lustre: server umount lustre-MDT0001 complete [ 6570.698947] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6573.222376] Key type lgssc unregistered [ 6573.490135] LNet: 237850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6573.500764] LNetError: 237850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6573.536489] LNet: Removed LNI 192.168.203.150@tcp [ 6574.369775] Key type .llcrypt unregistered [ 6574.374049] Key type ._llcrypt unregistered [ 6593.266352] Key type ._llcrypt registered [ 6593.268262] Key type .llcrypt registered [ 6593.344960] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6594.384235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6594.427471] alg: No test for adler32 (adler32-zlib) [ 6595.597876] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6595.885106] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6597.585582] Key type lgssc registered [ 6598.777722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6607.836252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6607.852441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.264197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6612.678606] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6618.594299] LustreError: 239292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.608417] LustreError: 239292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6618.996674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.079956] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6619.114419] Lustre: 239282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6619.121519] Lustre: Permanently deactivating lustre-MDT0001 [ 6619.144206] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6619.412287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6622.427815] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6622.434960] LustreError: 239323:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6622.448183] LustreError: 239323:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6622.465785] LustreError: 239323:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 6622.746496] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6629.008238] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6633.121330] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6637.481799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.714809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.361799] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6643.874329] LustreError: 241133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.899146] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6648.345814] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.314773] LustreError: 241135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.175874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.371736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.383554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6654.393356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6654.415471] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6657.460304] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6662.407721] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6665.686905] Lustre: Permanently reactivating lustre-MDT0001 [ 6665.699260] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6665.704159] Lustre: Skipped 2 previous similar messages [ 6668.068257] Lustre: lustre-MDT0001: Client 4410bca8-7926-4c2a-9f82-6d17923bd98a (at 192.168.203.50@tcp) reconnecting [ 6672.909821] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6673.933400] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6677.899342] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6678.050704] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6681.295141] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6681.479466] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6683.288305] Lustre: Permanently deactivating lustre-MDT0001 [ 6683.306219] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6683.314432] Lustre: Skipped 2 previous similar messages [ 6689.846160] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6693.500750] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6693.505549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.515702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6694.314133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6694.317960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.330356] Lustre: Skipped 2 previous similar messages [ 6699.491599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6699.502601] Lustre: Skipped 4 previous similar messages [ 6699.782545] Lustre: server umount lustre-MDT0000 complete [ 6703.242281] LustreError: 239271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592840 with bad export cookie 7371327160067149393 [ 6703.243254] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.259320] LustreError: 239271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6703.546243] Lustre: server umount lustre-MDT0001 complete [ 6712.782988] Lustre: server umount lustre-OST0000 complete [ 6722.773652] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:20:58 (1788592858) [ 6731.020271] Lustre: server umount lustre-OST0001 complete [ 6736.096942] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6744.195930] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6773.805976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.122665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6774.129628] Lustre: Skipped 2 previous similar messages [ 6774.146852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6774.230914] Lustre: lustre-MDT0000: new disk, initializing [ 6774.316754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.331829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6778.299326] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6788.127225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.277055] Lustre: 247546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6788.394396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6788.401821] Lustre: Skipped 1 previous similar message [ 6788.536769] Lustre: lustre-MDT0001: new disk, initializing [ 6788.651828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6788.684656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6788.692783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6792.628330] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6801.442655] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6807.459252] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6813.751038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6814.047975] Lustre: lustre-OST0000: new disk, initializing [ 6814.058236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6814.069689] Lustre: 249513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6814.180790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.052125] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6822.470696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6822.480837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6822.566662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6828.482925] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.302141] LustreError: 247554:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6838.530759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6838.733583] Lustre: lustre-OST0001: new disk, initializing [ 6838.750032] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6838.762356] Lustre: 250627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6840.344143] LustreError: 247554:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6840.365797] LustreError: 247553:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6845.062657] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6847.578928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6847.595974] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6847.686261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6850.392136] LustreError: 247553:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6853.877974] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6855.224446] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6858.214974] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6858.215019] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.233580] LustreError: Skipped 1 previous similar message [ 6858.241174] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6858.247460] Lustre: Skipped 1 previous similar message [ 6858.265435] Lustre: Skipped 1 previous similar message [ 6863.235637] Lustre: server umount lustre-OST0001 complete [ 6863.339868] LustreError: 251483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.356682] LustreError: 251483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6868.464971] LustreError: 251483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6868.483965] LustreError: 251483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6878.697914] LustreError: 249496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.720456] LustreError: 249496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6883.813294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.818312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.821923] Lustre: Skipped 1 previous similar message [ 6883.833171] Lustre: Skipped 2 previous similar messages [ 6885.642770] Lustre: server umount lustre-OST0000 complete [ 6888.931690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.942178] Lustre: Skipped 1 previous similar message [ 6894.051481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.062566] Lustre: Skipped 3 previous similar messages [ 6894.646424] Lustre: server umount lustre-MDT0000 complete [ 6897.806272] LustreError: 247540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593035 with bad export cookie 7371327160067152214 [ 6897.813164] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6897.815927] LustreError: 247540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6904.280527] Lustre: server umount lustre-MDT0001 complete [ 6915.408158] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6918.144423] Key type lgssc unregistered [ 6918.443841] LNet: 252662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6918.452010] LNetError: 252662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6918.470072] LNet: Removed LNI 192.168.203.150@tcp [ 6919.162139] Key type .llcrypt unregistered [ 6919.163853] Key type ._llcrypt unregistered [ 6953.315668] Key type ._llcrypt registered [ 6953.318628] Key type .llcrypt registered [ 6953.426743] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6954.491634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6954.608442] alg: No test for adler32 (adler32-zlib) [ 6955.693719] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6956.045738] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6957.728226] Key type lgssc registered [ 6958.995764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6965.470268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6965.495656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.797951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6966.837140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6966.978770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.440224] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6978.397978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.653685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6978.686146] Lustre: 254866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6978.842901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6982.896651] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6990.649199] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.832424] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7002.613281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.716792] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7002.857943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.445802] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7008.403428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7013.746306] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7065.060796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7065.061071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.069780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7065.089310] Lustre: Skipped 1 previous similar message [ 7065.111781] Lustre: Skipped 1 previous similar message [ 7070.184663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.196900] Lustre: Skipped 1 previous similar message [ 7071.101917] Lustre: server umount lustre-OST0000 complete [ 7075.042827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7086.688835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7086.919721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.696339] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7095.143704] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7095.156283] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7095.163300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7098.883375] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7141.347212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.352900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7141.374854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7141.381639] Lustre: Skipped 1 previous similar message [ 7143.203404] Lustre: server umount lustre-OST0000 complete [ 7148.001790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7148.016591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.035014] Lustre: Skipped 1 previous similar message [ 7148.040593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.585959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.013645] Lustre: server umount lustre-MDT0000 complete [ 7156.089237] LustreError: 254858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593293 with bad export cookie 15947101022103876432 [ 7156.097938] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.102393] LustreError: 254858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7156.442406] Lustre: server umount lustre-MDT0001 complete [ 7166.349815] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7169.011582] Key type lgssc unregistered [ 7169.327876] LNet: 260286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7169.344404] LNetError: 260286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7169.362561] LNet: Removed LNI 192.168.203.150@tcp [ 7170.123722] Key type .llcrypt unregistered [ 7170.126243] Key type ._llcrypt unregistered [ 7190.950598] Key type ._llcrypt registered [ 7190.952688] Key type .llcrypt registered [ 7191.062629] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7191.930130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7192.025108] alg: No test for adler32 (adler32-zlib) [ 7193.075821] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7193.311069] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7194.968315] Key type lgssc registered [ 7196.095827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.980669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.996598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.446119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.904878] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7214.566766] LustreError: 261679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.583371] LustreError: 261679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7216.961036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.307085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7221.626986] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7227.885733] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7232.942589] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7238.583730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.852442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.658817] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7249.149113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7251.116151] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7269.857127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.859858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7269.868251] Lustre: Skipped 1 previous similar message [ 7269.891809] Lustre: Skipped 1 previous similar message [ 7274.978150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7275.714737] Lustre: server umount lustre-OST0000 complete [ 7280.104265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.110448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.121583] Lustre: Skipped 1 previous similar message [ 7280.135248] Lustre: Skipped 1 previous similar message [ 7285.218046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.224180] Lustre: Skipped 2 previous similar messages [ 7290.345470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7290.356541] Lustre: Skipped 1 previous similar message [ 7293.408122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7293.590434] Lustre: server umount lustre-MDT0000 complete [ 7295.462076] LustreError: 261678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.493337] LustreError: 261678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7297.114775] LustreError: 261658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593434 with bad export cookie 6334873880166699860 [ 7297.115777] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7297.131366] LustreError: 261658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7297.688087] Lustre: server umount lustre-MDT0001 complete [ 7307.679981] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7310.429706] Key type lgssc unregistered [ 7310.796602] LNet: 265988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7310.815404] LNetError: 265988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7310.843556] LNet: Removed LNI 192.168.203.150@tcp [ 7311.741939] Key type .llcrypt unregistered [ 7311.746958] Key type ._llcrypt unregistered [ 7328.357955] Key type ._llcrypt registered [ 7328.360389] Key type .llcrypt registered [ 7328.460330] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7329.647662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7329.670501] alg: No test for adler32 (adler32-zlib) [ 7330.779853] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7330.952498] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7332.576226] Key type lgssc registered [ 7333.811373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7342.279788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7342.321200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.667933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7348.550561] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7353.828686] LustreError: 267209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.844136] LustreError: 267209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7357.326759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.712675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7362.436160] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7368.945101] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7374.539393] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7380.767875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.141625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.263272] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7389.426535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7393.814478] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7404.000908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7404.014268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.038598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.025848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7405.040464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7405.047543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.074328] Lustre: Skipped 1 previous similar message [ 7409.114783] Lustre: server umount lustre-OST0000 complete [ 7414.241327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7414.250294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.260792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.686602] Lustre: server umount lustre-MDT0000 complete [ 7420.387248] LustreError: 267206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.394215] LustreError: 267206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7422.067580] LustreError: 267190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593559 with bad export cookie 11924794861660054474 [ 7422.087586] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7422.094958] LustreError: 267190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7422.670548] Lustre: server umount lustre-MDT0001 complete [ 7434.634526] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7437.715737] Key type lgssc unregistered [ 7437.977149] LNet: 270884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7437.987512] LNetError: 270884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7439.022727] LNet: Removed LNI 192.168.203.150@tcp [ 7439.895682] Key type .llcrypt unregistered [ 7439.897480] Key type ._llcrypt unregistered [ 7461.937581] Key type ._llcrypt registered [ 7461.941145] Key type .llcrypt registered [ 7462.072787] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7462.879343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7462.930615] alg: No test for adler32 (adler32-zlib) [ 7464.034924] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7464.281597] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7465.912168] Key type lgssc registered [ 7467.149800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7473.762536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7473.788539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.265178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7479.136931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7485.418360] LustreError: 272295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.439413] LustreError: 272295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7486.858164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.323947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7491.147527] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7496.479916] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7501.190652] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7507.134192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7507.382316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.513923] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7519.689540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7519.812317] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7543.264881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7543.274528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.288370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.320745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.321417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.345386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7547.710738] Lustre: server umount lustre-OST0000 complete [ 7554.022705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7554.039587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.052895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7554.057478] Lustre: Skipped 1 previous similar message [ 7560.673810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.680740] Lustre: Skipped 3 previous similar messages [ 7565.280611] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7565.452201] Lustre: server umount lustre-MDT0000 complete [ 7565.804673] LustreError: 272291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.815409] LustreError: 272291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7568.813234] LustreError: 272277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593706 with bad export cookie 11869897256918561484 [ 7568.820850] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7568.832450] LustreError: 272277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7569.259961] Lustre: server umount lustre-MDT0001 complete [ 7579.283196] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7581.794391] Key type lgssc unregistered [ 7582.024127] LNet: 276602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.048114] LNetError: 276602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7582.066779] LNet: Removed LNI 192.168.203.150@tcp [ 7582.839348] Key type .llcrypt unregistered [ 7582.841636] Key type ._llcrypt unregistered [ 7601.150843] Key type ._llcrypt registered [ 7601.153215] Key type .llcrypt registered [ 7601.298308] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7602.542955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7602.870047] alg: No test for adler32 (adler32-zlib) [ 7604.016991] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7604.355894] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7606.112159] Key type lgssc registered [ 7607.892817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7618.117244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7618.151282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7619.724378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7624.484762] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7630.311158] LustreError: 277836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.328659] LustreError: 277836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7632.462195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.859950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7637.544770] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7645.310695] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7651.194912] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7658.182211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7658.481233] LustreError: 279679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.657241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.173345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7665.689833] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7673.883826] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7689.700931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7689.712968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7689.748625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7690.732309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7690.740854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7690.758513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.550961] Lustre: server umount lustre-OST0000 complete [ 7699.432926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7699.444275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.477581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7699.487591] Lustre: Skipped 1 previous similar message [ 7702.696411] Lustre: server umount lustre-MDT0000 complete [ 7705.995492] LustreError: 277803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593843 with bad export cookie 913092907881192721 [ 7706.004471] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.322586] Lustre: server umount lustre-MDT0001 complete [ 7715.573397] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7717.941752] Key type lgssc unregistered [ 7718.250796] LNet: 281535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7718.271796] LNetError: 281535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7718.301744] LNet: Removed LNI 192.168.203.150@tcp [ 7719.041721] Key type .llcrypt unregistered [ 7719.044271] Key type ._llcrypt unregistered [ 7731.388622] Key type ._llcrypt registered [ 7731.393415] Key type .llcrypt registered [ 7731.517279] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7750.736945] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7765.808180] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7766.783799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7766.794916] alg: No test for adler32 (adler32-zlib) [ 7767.782294] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7767.938607] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7769.568164] Key type lgssc registered [ 7770.882213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7809.789054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7809.848634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.130443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7811.159996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7811.249866] Lustre: lustre-MDT0000: new disk, initializing [ 7811.361658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.386932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7815.352744] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7826.051319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.196897] Lustre: 285085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7826.246958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7826.250721] Lustre: Skipped 1 previous similar message [ 7826.303832] Lustre: lustre-MDT0001: new disk, initializing [ 7826.372911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7826.410715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7826.429316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7830.430373] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7839.107847] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7844.143888] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7850.658955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.944369] Lustre: lustre-OST0000: new disk, initializing [ 7850.950625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7850.961957] Lustre: 287055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7851.027284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7857.376317] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7859.798112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7859.823823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7859.881584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7867.197245] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7871.901939] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7872.098653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7877.029332] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7877.305536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7880.680648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7880.681383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.705431] LustreError: Skipped 1 previous similar message [ 7880.706183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7880.725370] Lustre: Skipped 1 previous similar message [ 7885.397432] Lustre: server umount lustre-OST0000 complete [ 7890.915103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.946888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7890.958374] Lustre: Skipped 2 previous similar messages [ 7895.457175] Lustre: server umount lustre-MDT0000 complete [ 7899.313607] LustreError: 287054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594036 with bad export cookie 7137444663226224652 [ 7899.333546] LustreError: 287054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7899.341366] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7899.618402] Lustre: server umount lustre-MDT0001 complete [ 7907.935704] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:40:43 (1788594043) [ 7914.144296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.644154] LustreError: 289465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.661369] LustreError: 289465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7914.705325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.746790] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7920.098588] LustreError: 289466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7925.221238] LustreError: 289465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.754091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.129786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7933.160336] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7940.833577] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7946.337390] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7953.014630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.280142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.119792] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7968.569701] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.001895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.008752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.027262] Lustre: Skipped 1 previous similar message [ 7980.033476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.054464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.054993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7984.335770] Lustre: server umount lustre-OST0000 complete [ 7990.755260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7990.768666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.786263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.790472] Lustre: Skipped 1 previous similar message [ 7994.552480] Lustre: server umount lustre-MDT0000 complete [ 7997.413978] LustreError: 290194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.444470] LustreError: 290194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7998.731879] LustreError: 289447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594136 with bad export cookie 7137444663226225541 [ 7998.737453] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.752687] LustreError: 289447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7999.244181] Lustre: server umount lustre-MDT0001 complete [ 8010.476275] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8013.459470] Key type lgssc unregistered [ 8013.825268] LNet: 293001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8013.836066] LNetError: 293001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8013.862794] LNet: Removed LNI 192.168.203.150@tcp [ 8014.895522] Key type .llcrypt unregistered [ 8014.898165] Key type ._llcrypt unregistered [ 8043.434273] Key type ._llcrypt registered [ 8043.436687] Key type .llcrypt registered [ 8043.527664] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8058.279789] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8059.133120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8059.161183] alg: No test for adler32 (adler32-zlib) [ 8060.382272] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8060.669639] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8062.400161] Key type lgssc registered [ 8063.522620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8107.257481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8107.285494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.623792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8108.680824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8108.805429] Lustre: lustre-MDT0000: new disk, initializing [ 8108.958298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8108.993394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8113.868462] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8126.104722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.306822] Lustre: 296825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8126.404576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8126.413141] Lustre: Skipped 1 previous similar message [ 8126.483727] Lustre: lustre-MDT0001: new disk, initializing [ 8126.640338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8126.689098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8126.699148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8131.886758] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8142.021551] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.697636] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8154.442516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.891641] Lustre: lustre-OST2710: new disk, initializing [ 8154.898514] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8154.909120] Lustre: 298795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8155.028720] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8161.152768] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8163.457219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8163.484016] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8163.602894] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8171.379624] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8178.191201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8178.370332] Lustre: lustre-OST03e8: new disk, initializing [ 8178.377985] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8178.387843] Lustre: 299858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8178.512175] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8184.471270] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8185.946129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8185.954570] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8186.066632] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8192.631957] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8199.792409] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8200.092822] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8205.120178] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8205.323975] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8210.346786] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8210.551403] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8215.550093] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8215.837587] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8222.185721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8222.192366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.214769] Lustre: Skipped 3 previous similar messages [ 8222.226476] Lustre: Skipped 2 previous similar messages [ 8222.998633] Lustre: server umount lustre-MDT0000 complete [ 8226.941635] LustreError: 296819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594364 with bad export cookie 204636475183433429 [ 8226.953746] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8226.953838] LustreError: 296819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8227.299204] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.383669] Lustre: server umount lustre-MDT0001 complete [ 8234.084499] Lustre: server umount lustre-OST2710 complete [ 8237.696711] Lustre: server umount lustre-OST03e8 complete [ 8258.386946] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8269.376854] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8302.798115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.137564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8303.172572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8303.256821] Lustre: lustre-MDT0000: new disk, initializing [ 8303.373209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.392696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8308.461750] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8318.592106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.708327] Lustre: 305105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8318.806598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8318.812111] Lustre: Skipped 1 previous similar message [ 8318.900367] Lustre: lustre-MDT0001: new disk, initializing [ 8319.027027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8319.120984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8319.142922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8323.893289] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8333.395720] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8339.176525] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8345.614724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.908322] Lustre: lustre-OST0000: new disk, initializing [ 8345.912324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8345.918331] Lustre: 307075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8346.015323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.801417] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8353.349029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8353.358330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8353.441732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8361.836034] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8366.438018] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8366.652921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8370.887637] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8371.104954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8374.246769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.247264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.249734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.249741] Lustre: Skipped 1 previous similar message [ 8374.301422] Lustre: Skipped 2 previous similar messages [ 8379.165954] Lustre: server umount lustre-OST0000 complete [ 8384.481623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8384.500768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8384.508108] Lustre: Skipped 1 previous similar message [ 8389.356883] Lustre: server umount lustre-MDT0000 complete [ 8389.603417] LustreError: 307705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.619310] LustreError: 307705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8392.934251] LustreError: 305098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594530 with bad export cookie 204636475183435102 [ 8392.937026] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.957061] LustreError: 305098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8393.421862] Lustre: server umount lustre-MDT0001 complete [ 8401.555943] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:48:57 (1788594537) [ 8403.540410] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8405.240545] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:49:00 (1788594540) [ 8408.935979] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8437.310903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.573772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8437.608477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8437.874296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.069161] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8451.842217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.991514] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8452.040786] Lustre: 310934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8452.121721] LustreError: 310944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.882427] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8477.747627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.000803] Lustre: Denying initial registration attempt from nid 192.168.203.150@tcp, specified as failover [ 8478.014250] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8478.021724] LustreError: 312386:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8478.032735] LustreError: 312386:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8478.048477] LustreError: 312386:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8478.128946] Lustre: server umount lustre-OST0000 complete [ 8478.132968] LustreError: 312386:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8488.421300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8488.423396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8488.431174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8488.442876] Lustre: Skipped 2 previous similar messages [ 8491.530412] Lustre: server umount lustre-MDT0000 complete [ 8493.537413] LustreError: 310944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8493.571337] LustreError: 310944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8495.147247] LustreError: 310925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594632 with bad export cookie 204636475183435991 [ 8495.149651] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.161104] LustreError: 310925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8495.450865] Lustre: server umount lustre-MDT0001 complete [ 8505.270474] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8507.547937] Key type lgssc unregistered [ 8507.833472] LNet: 313506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8507.840365] LNetError: 313506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8507.865621] LNet: Removed LNI 192.168.203.150@tcp [ 8508.617495] Key type .llcrypt unregistered [ 8508.619259] Key type ._llcrypt unregistered [ 8519.071910] Key type ._llcrypt registered [ 8519.077039] Key type .llcrypt registered [ 8519.253110] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8520.504725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8520.544766] alg: No test for adler32 (adler32-zlib) [ 8521.763994] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8521.986131] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8523.864140] Key type lgssc registered [ 8525.105115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8571.616638] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8579.794509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8579.817638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.076581] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8581.093493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8581.219611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.863566] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8594.250805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.384788] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8594.424128] Lustre: 316282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8594.683032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8599.044644] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8606.127634] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8611.752801] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8631.289152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8631.556430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8631.961917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8638.248988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8644.954367] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8649.186447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.197496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.217045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8649.701074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.707039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.724472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.358104] Lustre: server umount lustre-OST0000 complete [ 8659.937979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.955779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.962829] Lustre: Skipped 1 previous similar message [ 8663.338128] Lustre: server umount lustre-MDT0000 complete [ 8666.870164] LustreError: 316275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594804 with bad export cookie 5762809633866239970 [ 8666.875793] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.881547] LustreError: 316275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8667.218929] Lustre: server umount lustre-MDT0001 complete [ 8678.730439] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8681.397586] Key type lgssc unregistered [ 8681.684250] LNet: 319857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8681.697925] LNetError: 319857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8681.716378] LNet: Removed LNI 192.168.203.150@tcp [ 8682.548323] Key type .llcrypt unregistered [ 8682.553856] Key type ._llcrypt unregistered [ 8706.222233] Key type ._llcrypt registered [ 8706.225980] Key type .llcrypt registered [ 8706.356397] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8707.454099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8707.643473] alg: No test for adler32 (adler32-zlib) [ 8708.793318] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8709.086661] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8710.728635] Key type lgssc registered [ 8711.776266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8719.183687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8719.232628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8720.801287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8725.474433] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8731.109205] LustreError: 321271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8731.131493] LustreError: 321271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8733.863075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8734.282432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8738.852555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8746.059345] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8751.638176] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8759.218541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8759.543089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.789642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8771.272616] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8790.497819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8790.506067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.537756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.330328] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 8794.594698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.596852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.599172] Lustre: Skipped 1 previous similar message [ 8794.605021] Lustre: Skipped 1 previous similar message [ 8798.417360] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 8798.429063] Lustre: Skipped 1 previous similar message [ 8803.537177] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 8803.557125] Lustre: Skipped 3 previous similar messages [ 8804.320109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8804.457985] Lustre: server umount lustre-MDT0000 complete [ 8804.839842] LustreError: 322274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.864159] LustreError: 322274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8807.581433] LustreError: 321252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594945 with bad export cookie 11678804259923392055 [ 8807.610151] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8808.651809] LustreError: 322559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.953754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8812.963847] LustreError: 321265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8812.965080] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8812.987324] LustreError: 321265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8812.991641] Lustre: Skipped 2 previous similar messages [ 8813.865362] Lustre: server umount lustre-MDT0001 complete [ 8819.103586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8826.918261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8833.513430] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa2137298f20d948a [ 8833.534988] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 8833.853687] LustreError: 323103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8833.880264] LustreError: 323103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8833.942941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.025366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8838.250484] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8843.176161] LustreError: 323104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8843.189548] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8843.205921] LustreError: 323104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8846.099452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8846.400950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8851.123368] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8851.445498] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8858.461876] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8859.946171] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8865.749266] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8867.348088] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8874.980408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8874.993719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.015205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8875.025884] Lustre: Skipped 1 previous similar message [ 8878.389490] Lustre: server umount lustre-OST0000 complete [ 8887.271769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.275837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8887.288108] Lustre: Skipped 1 previous similar message [ 8888.868424] Lustre: server umount lustre-MDT0000 complete [ 8892.398917] LustreError: 325621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.430592] LustreError: 325621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8893.014731] LustreError: 321250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595030 with bad export cookie 11678804259923399818 [ 8893.016805] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.029769] LustreError: 321250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8893.535230] Lustre: server umount lustre-MDT0001 complete [ 8904.307541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8907.901543] Key type lgssc unregistered [ 8908.383644] LNet: 327560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8908.399561] LNetError: 327560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8908.416040] LNet: Removed LNI 192.168.203.150@tcp [ 8909.390174] Key type .llcrypt unregistered [ 8909.401903] Key type ._llcrypt unregistered [ 8931.767393] Key type ._llcrypt registered [ 8931.770841] Key type .llcrypt registered [ 8931.906372] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8932.917847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8932.994017] alg: No test for adler32 (adler32-zlib) [ 8934.148229] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8934.445712] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8936.128220] Key type lgssc registered [ 8937.463334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8946.144561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8946.161399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8947.661350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8952.067156] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8957.921820] LustreError: 328974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.951997] LustreError: 328974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8960.759856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.279765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8966.280099] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8974.045392] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8979.333523] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8981.901173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.907672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8981.911492] Lustre: Skipped 1 previous similar message [ 8981.923348] Lustre: Skipped 1 previous similar message [ 8986.022379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.032416] Lustre: Skipped 1 previous similar message [ 8991.202418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.206013] Lustre: Skipped 1 previous similar message [ 8996.330199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8996.333337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8996.370695] Lustre: Skipped 1 previous similar message [ 8996.729531] Lustre: server umount lustre-MDT0000 complete [ 9000.657115] LustreError: 328955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595138 with bad export cookie 4949437655295812244 [ 9000.667090] LustreError: 328955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9000.677484] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.950688] Lustre: server umount lustre-MDT0001 complete [ 9014.877596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.204345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9015.264072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9015.487140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.748388] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9029.339058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.535344] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9029.608335] Lustre: 331564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9030.183981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9036.177571] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9045.376603] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9051.089362] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9061.127075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.335672] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9061.343727] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9061.358024] LustreError: 333448:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9061.371951] LustreError: 333448:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9061.383672] LustreError: 333448:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9061.467637] Lustre: server umount lustre-OST0000 complete [ 9061.478977] LustreError: 333448:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9070.836483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9071.016132] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9071.232747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9077.256961] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9080.533919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9085.529919] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9092.784076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.858703] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9092.933699] Lustre: lustre-OST0001: new disk, initializing [ 9092.937309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9092.942854] Lustre: 334988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9093.041338] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9098.945883] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9099.359968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9099.369603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9099.468143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9105.384777] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9109.985940] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9109.986418] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9109.999507] LustreError: Skipped 1 previous similar message [ 9110.004639] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9110.027285] Lustre: Skipped 1 previous similar message [ 9118.187754] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9118.196063] Lustre: Skipped 4 previous similar messages [ 9122.272099] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9122.493903] Lustre: server umount lustre-OST0001 complete [ 9123.306274] LustreError: 333915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9123.325099] LustreError: 333915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9126.881349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9126.883554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.896476] LustreError: Skipped 1 previous similar message [ 9126.930887] Lustre: Skipped 1 previous similar message [ 9128.418576] LustreError: 333915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.464703] LustreError: 333915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9133.539658] LustreError: 335371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.565307] LustreError: 335371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9138.662850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9138.671624] Lustre: Skipped 7 previous similar messages [ 9138.675687] LustreError: 335371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.692978] LustreError: 335371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9141.216138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9141.489546] Lustre: server umount lustre-OST0000 complete [ 9147.873166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9147.881450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.175479] Lustre: server umount lustre-MDT0000 complete [ 9154.017956] LustreError: 331575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.027538] LustreError: 331575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9154.229921] LustreError: 331556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595291 with bad export cookie 4949437655295812797 [ 9154.240553] LustreError: 331556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9154.241254] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.509874] Lustre: server umount lustre-MDT0001 complete [ 9163.493761] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9165.845318] Key type lgssc unregistered [ 9166.089296] LNet: 336958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9166.103616] LNetError: 336958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9166.123439] LNet: Removed LNI 192.168.203.150@tcp [ 9166.839067] Key type .llcrypt unregistered [ 9166.842924] Key type ._llcrypt unregistered [ 9229.405225] Key type ._llcrypt registered [ 9229.410344] Key type .llcrypt registered [ 9229.519145] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9244.504766] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9245.782613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9245.798671] alg: No test for adler32 (adler32-zlib) [ 9247.075902] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9247.338247] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9249.048150] Key type lgssc registered [ 9250.387857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9282.861119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9282.908192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.118209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9284.151390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9284.262861] Lustre: lustre-MDT0000: new disk, initializing [ 9284.348300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.370402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9289.095499] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9298.906558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.962589] Lustre: 341886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9299.014936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9299.018607] Lustre: Skipped 1 previous similar message [ 9299.084917] Lustre: lustre-MDT0001: new disk, initializing [ 9299.176210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.212535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9299.228525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9303.233324] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9312.817795] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9318.385849] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9324.856094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9325.141113] Lustre: lustre-OST0000: new disk, initializing [ 9325.145470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9325.151564] Lustre: 343854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9325.247134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.540487] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9331.802054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9331.824880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9331.964333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9340.562310] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9345.999792] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9346.282130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9352.234630] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9352.596459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9357.794054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9357.800961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9357.804311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9357.808331] Lustre: Skipped 1 previous similar message [ 9357.822602] Lustre: Skipped 1 previous similar message [ 9360.942514] Lustre: server umount lustre-OST0000 complete [ 9365.987113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9365.994942] LustreError: Skipped 1 previous similar message [ 9365.998701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9366.012158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9368.035756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9368.051736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9370.927197] Lustre: server umount lustre-MDT0000 complete [ 9374.863251] LustreError: 342681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595512 with bad export cookie 18287765600473335196 [ 9374.868839] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9374.871274] LustreError: 342681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9375.458455] Lustre: server umount lustre-MDT0001 complete [ 9383.594037] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:05:19 (1788595519) [ 9389.572588] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:05:25 (1788595525) [ 9396.667133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.153135] LustreError: 346610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9397.173820] LustreError: 346610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9397.216746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.304096] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9402.339704] LustreError: 346611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.459779] LustreError: 346610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.883400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9409.232764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.300908] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9420.289571] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9425.327235] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9431.422228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9431.818216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9437.889872] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9444.241114] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9451.647587] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9453.162599] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9454.615389] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9456.074073] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9457.814507] Lustre: Failing over lustre-MDT0000 [ 9458.136733] Lustre: server umount lustre-MDT0000 complete [ 9460.705338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9460.714776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9460.732456] LustreError: 347338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9467.593920] LustreError: 346606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9467.604470] LustreError: 346606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9475.758027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.886970] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9476.255512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.338884] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9477.856058] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9480.281537] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9481.715307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9481.765988] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9481.844251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9489.723180] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9491.349797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9494.939440] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9496.799879] Lustre: Failing over lustre-MDT0000 [ 9497.040905] Lustre: server umount lustre-MDT0000 complete [ 9497.064505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9497.077217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.104644] Lustre: Skipped 4 previous similar messages [ 9497.112270] LustreError: 347928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.138291] LustreError: 347928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9512.913288] Lustre: 340111:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595634/real 1788595634] req@000000009c4ec786 x1875478193460224/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788595650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9512.949824] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9513.703593] LustreError: 347928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9513.719184] LustreError: 347928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9515.111110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.189246] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfdcb31ebbfd25031 [ 9523.205159] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 9523.213260] Lustre: Skipped 2 previous similar messages [ 9523.509966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9523.568714] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9523.917711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9527.783311] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9528.833430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9528.864468] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9528.912547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9536.611435] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9537.917176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9544.162967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.167263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9544.168767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9544.168773] Lustre: Skipped 1 previous similar message [ 9544.180176] Lustre: Skipped 1 previous similar message [ 9549.281715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9549.292140] Lustre: Skipped 4 previous similar messages [ 9549.525428] Lustre: server umount lustre-MDT0000 complete [ 9553.251417] LustreError: 346590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595690 with bad export cookie 18287765600473337905 [ 9553.253917] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9553.257936] LustreError: 346590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9553.875875] Lustre: server umount lustre-MDT0001 complete [ 9568.049608] Lustre: server umount lustre-OST0000 complete [ 9582.163762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.673581] LustreError: 352947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9582.695077] LustreError: 352947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9586.447625] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9595.527742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9596.053846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9596.059873] Lustre: Skipped 1 previous similar message [ 9600.774261] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9609.074273] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9614.779145] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9621.869476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9628.270346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9628.412152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9634.708376] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9651.172745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.191996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9651.196305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9651.204877] Lustre: Skipped 1 previous similar message [ 9655.922972] Lustre: server umount lustre-OST0000 complete [ 9670.627868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.633160] Lustre: Skipped 5 previous similar messages [ 9674.208743] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9674.385086] Lustre: server umount lustre-MDT0000 complete [ 9675.749754] LustreError: 352947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.768032] LustreError: 352947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9677.342732] LustreError: 354792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595814 with bad export cookie 18287765600473338654 [ 9677.349912] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9677.350479] LustreError: 354792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9677.633808] Lustre: server umount lustre-MDT0001 complete [ 9689.341463] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9692.982736] Key type lgssc unregistered [ 9693.410035] LNet: 356504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9693.423750] LNetError: 356504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9693.443860] LNet: Removed LNI 192.168.203.150@tcp [ 9694.855471] Key type .llcrypt unregistered [ 9694.857973] Key type ._llcrypt unregistered [ 9719.407862] Key type ._llcrypt registered [ 9719.412586] Key type .llcrypt registered [ 9719.527590] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9720.383395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9720.480343] alg: No test for adler32 (adler32-zlib) [ 9721.576901] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9721.856608] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9723.633234] Key type lgssc registered [ 9724.649676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9732.174943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9732.192784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9733.529954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.938264] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9742.823417] LustreError: 358018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.843168] LustreError: 358018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9745.481896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.785512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9749.672873] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9756.458422] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9760.928763] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9767.491557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9767.843671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.700489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9778.116474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9781.753451] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9791.970303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9791.978068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9791.989276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.512615] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9793.514093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.525711] Lustre: Skipped 1 previous similar message [ 9793.547922] Lustre: Skipped 2 previous similar messages [ 9797.053647] Lustre: server umount lustre-MDT0000 complete [ 9798.637381] LustreError: 359084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.677256] LustreError: 359084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9800.306800] LustreError: 357998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595937 with bad export cookie 17262755800134128812 [ 9800.318242] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.319842] LustreError: 357998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9800.595207] Lustre: server umount lustre-MDT0001 complete [ 9814.213312] Lustre: server umount lustre-OST0000 complete [ 9826.421259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.905069] LustreError: 361729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.999757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.345967] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9832.419241] LustreError: 361730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.539295] LustreError: 361729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.197764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.594886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.176957] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9852.267728] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9858.739877] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9865.620843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9865.908766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.060458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9872.141705] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9879.297068] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9890.786472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9890.797128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.811114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9894.884096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9894.884395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.127548] Lustre: server umount lustre-MDT0000 complete [ 9900.009139] LustreError: 361725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9900.033423] LustreError: 361725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9900.721593] LustreError: 361709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596038 with bad export cookie 17262755800134129932 [ 9900.731314] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9900.743576] LustreError: 361709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9901.155754] Lustre: server umount lustre-MDT0001 complete [ 9914.990518] Lustre: server umount lustre-OST0000 complete [ 9927.764072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.073526] LustreError: 365638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.131467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9932.000404] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9939.483305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.879860] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9950.598778] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9955.813078] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9961.166790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9961.548144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.554030] Lustre: Skipped 1 previous similar message [ 9966.708401] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9972.865135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9974.878329] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9980.385165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9980.396786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9980.414396] Lustre: Skipped 1 previous similar message [ 9980.422691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9980.428536] Lustre: Skipped 2 previous similar messages [ 9983.457593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9985.871465] Lustre: server umount lustre-OST0000 complete [ 9991.146205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9991.161990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9991.185032] Lustre: Skipped 1 previous similar message [ 9991.198065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9991.211187] Lustre: Skipped 2 previous similar messages [ 9996.393700] Lustre: server umount lustre-MDT0000 complete [ 9998.820071] LustreError: 365638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.847035] LustreError: 365638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10000.022334] LustreError: 365620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596137 with bad export cookie 17262755800134130947 [10000.035573] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.335327] Lustre: server umount lustre-MDT0001 complete [10010.237534] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10013.044797] Key type lgssc unregistered [10013.350244] LNet: 369168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10013.358593] LNetError: 369168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10013.375857] LNet: Removed LNI 192.168.203.150@tcp [10014.300161] Key type .llcrypt unregistered [10014.306377] Key type ._llcrypt unregistered [10035.878969] Key type ._llcrypt registered [10035.881530] Key type .llcrypt registered [10036.011555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10036.953175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10037.096642] alg: No test for adler32 (adler32-zlib) [10038.120533] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [10038.370336] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10040.041334] Key type lgssc registered [10041.310766] Lustre: Echo OBD driver; http://www.lustre.org/ [10048.400119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10048.421654] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.430681] LustreError: 370587:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10048.445637] LustreError: 370587:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10048.454584] LustreError: 370587:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10048.463555] LustreError: 370587:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10048.474738] LustreError: 370587:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10055.828832] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.835575] LustreError: 370985:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10055.851553] LustreError: 370985:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10055.858514] LustreError: 370985:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10055.864437] LustreError: 370985:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10055.872713] LustreError: 370985:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10073.422114] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10076.414948] Key type lgssc unregistered [10076.690718] LNet: 371987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10076.705098] LNetError: 371987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10076.731707] LNet: Removed LNI 192.168.203.150@tcp [10077.469821] Key type .llcrypt unregistered [10077.477324] Key type ._llcrypt unregistered [10094.810947] Key type ._llcrypt registered [10094.813170] Key type .llcrypt registered [10094.921609] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [10109.866246] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10111.091231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10111.123481] alg: No test for adler32 (adler32-zlib) [10112.175037] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [10112.319618] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10113.952194] Key type lgssc registered [10115.063339] Lustre: Echo OBD driver; http://www.lustre.org/ [10147.092432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10147.157063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.403722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10148.433039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10148.493980] Lustre: lustre-MDT0000: new disk, initializing [10148.656408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10148.681871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10152.142648] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10161.041512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.128408] Lustre: 375140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10161.175450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10161.178461] Lustre: Skipped 1 previous similar message [10161.246707] Lustre: lustre-MDT0001: new disk, initializing [10161.352265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10161.407993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10161.425264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10165.300443] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10174.425489] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.940853] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10187.668240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10187.945330] Lustre: lustre-OST0000: new disk, initializing [10187.948972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10187.957687] Lustre: 377107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10188.100513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10193.654405] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10195.044079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10195.055559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10195.166569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10201.503776] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10205.296575] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10205.551175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10209.415846] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10209.648269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10215.907510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10215.912706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.916153] LustreError: Skipped 1 previous similar message [10215.924482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10215.942934] Lustre: Skipped 1 previous similar message [10215.964996] Lustre: Skipped 1 previous similar message [10217.402466] Lustre: server umount lustre-OST0000 complete [10223.073489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10223.083155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10223.103739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10226.146276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10226.150610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10227.587517] Lustre: server umount lustre-MDT0000 complete [10230.576582] LustreError: 375934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596367 with bad export cookie 13193269776307531879 [10230.580480] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.584976] LustreError: 375934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10230.840222] Lustre: server umount lustre-MDT0001 complete [10238.468332] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10240.320187] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:19:35 (1788596375) [10246.572840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.022985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10251.084128] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10252.258578] LustreError: 379567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.270400] LustreError: 379567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10257.377673] LustreError: 379566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.460054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.776077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10264.977370] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10273.039720] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10278.153387] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10284.049527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10284.361858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10289.693478] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10297.365446] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10303.583024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10303.707805] Lustre: lustre-OST0001: new disk, initializing [10303.711612] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10303.720287] Lustre: 382400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10303.824827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10309.712327] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10310.257494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10310.268807] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10310.374893] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10317.628449] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10325.729441] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [10325.738688] Lustre: Skipped 1 previous similar message [10325.992650] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10325.992687] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10328.418352] Lustre: server umount lustre-OST0001 complete [10330.829278] LustreError: 381389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.247484] LustreError: 381388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.261027] LustreError: 381388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10341.357354] LustreError: 381389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10341.379911] LustreError: 381389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10346.476619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10346.483839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10346.490672] Lustre: Skipped 2 previous similar messages [10346.494708] Lustre: Skipped 2 previous similar messages [10351.593664] LustreError: 383336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10351.613242] LustreError: 383336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10356.715158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10356.720979] Lustre: Skipped 3 previous similar messages [10359.265992] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10359.457447] Lustre: server umount lustre-OST0000 complete [10366.946604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10366.980949] Lustre: Skipped 1 previous similar message [10377.184182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10377.186360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.201422] Lustre: Skipped 5 previous similar messages [10377.391588] Lustre: server umount lustre-MDT0000 complete [10380.524724] LustreError: 379547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596517 with bad export cookie 13193269776307532768 [10380.527089] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10380.532684] LustreError: 379547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10387.052727] Lustre: server umount lustre-MDT0001 complete [10396.926065] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10399.525747] Key type lgssc unregistered [10399.838925] LNet: 384393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10399.859943] LNetError: 384393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10399.888879] LNet: Removed LNI 192.168.203.150@tcp [10400.699682] Key type .llcrypt unregistered [10400.703089] Key type ._llcrypt unregistered [10448.713655] Key type ._llcrypt registered [10448.717926] Key type .llcrypt registered [10448.836597] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10449.832475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10450.010396] alg: No test for adler32 (adler32-zlib) [10451.158456] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [10451.399566] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10453.048202] Key type lgssc registered [10453.943720] Lustre: Echo OBD driver; http://www.lustre.org/ [10460.458657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10460.481850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.709281] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10461.728406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10461.828682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10466.161613] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10473.187410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.299482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10473.342407] Lustre: 387188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10473.586821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10477.789585] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10484.487824] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10489.053682] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10490.849477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.852422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.865798] Lustre: Skipped 1 previous similar message [10495.972821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10495.988092] Lustre: Skipped 2 previous similar messages [10497.115788] Lustre: server umount lustre-MDT0000 complete [10501.204495] LustreError: 387181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596638 with bad export cookie 15682384865773221232 [10501.210649] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10501.212327] LustreError: 387181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10501.519295] Lustre: server umount lustre-MDT0001 complete [10510.372544] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:24:05 (1788596645) [10515.651434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.983423] LustreError: 389700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10515.995450] LustreError: 389700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10516.039468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10519.819402] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10521.063159] LustreError: 389701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10521.083736] LustreError: 389701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10526.177396] LustreError: 389700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.774028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.166769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10531.466692] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10538.480124] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10543.285705] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10549.442646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10549.695898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10549.910211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10555.563746] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10561.675855] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10569.357343] Lustre: Permanently deactivating lustre-OST0000 [10569.377931] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10578.938375] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10581.039103] LustreError: 392780:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10581.045655] LustreError: 392780:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10581.050914] LustreError: 392780:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10583.564877] LustreError: 392828:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10583.574398] LustreError: 392828:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10591.542530] Lustre: server umount lustre-OST0000 complete [10593.957121] LustreError: 393077:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10593.965768] LustreError: 393077:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10598.380710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10598.383672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10598.400532] Lustre: Skipped 1 previous similar message [10603.489917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10603.492695] Lustre: Skipped 1 previous similar message [10606.652430] Lustre: server umount lustre-MDT0000 complete [10608.610732] LustreError: 389697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.666028] LustreError: 389697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10610.431499] LustreError: 389682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596747 with bad export cookie 15682384865773221820 [10610.434343] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10610.450309] LustreError: 389682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10620.074572] Lustre: server umount lustre-MDT0001 complete [10628.462824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.926332] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10634.232204] LustreError: 394235:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10639.802432] Lustre: 394575:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.150@tcp [10641.818216] Lustre: server umount MGS complete [10651.681196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.357337] LustreError: 395299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.408418] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10652.422767] Lustre: Skipped 1 previous similar message [10652.498443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10656.832309] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10662.885536] LustreError: 395299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.894330] LustreError: 395299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10665.192226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.449606] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10670.590716] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10679.466139] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10686.802961] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.920663] Lustre: Permanently reactivating lustre-OST0000 [10688.937372] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10688.946257] Lustre: Skipped 2 previous similar messages [10715.104221] LustreError: 397033:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10721.735513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.016497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10722.022898] Lustre: Skipped 1 previous similar message [10727.417730] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10733.760601] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10740.710204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10740.711324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.723368] LustreError: Skipped 1 previous similar message [10740.736056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.743272] Lustre: Skipped 1 previous similar message [10744.296241] Lustre: server umount lustre-OST0000 complete [10747.362696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10747.371549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.380500] Lustre: Skipped 1 previous similar message [10750.950758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.959692] Lustre: Skipped 2 previous similar messages [10753.643507] Lustre: server umount lustre-MDT0000 complete [10756.081293] LustreError: 395295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.113985] LustreError: 395295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10756.892081] LustreError: 395280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596894 with bad export cookie 15682384865773222870 [10756.906026] LustreError: 395280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10756.909231] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10757.164667] Lustre: server umount lustre-MDT0001 complete [10765.809906] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10768.183047] Key type lgssc unregistered [10768.487418] LNet: 399663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10768.508578] LNetError: 399663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10768.524322] LNet: Removed LNI 192.168.203.150@tcp [10769.323713] Key type .llcrypt unregistered [10769.328776] Key type ._llcrypt unregistered [10784.101909] Key type ._llcrypt registered [10784.104324] Key type .llcrypt registered [10784.207547] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [10796.880965] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10797.923965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10798.046200] alg: No test for adler32 (adler32-zlib) [10799.121811] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [10799.432161] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10801.112213] Key type lgssc registered [10802.311132] Lustre: Echo OBD driver; http://www.lustre.org/ [10834.640095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10834.670963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.042366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10836.107668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10836.248965] Lustre: lustre-MDT0000: new disk, initializing [10836.378025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10836.419613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10842.111889] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10851.916470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.047381] Lustre: 402794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10852.108144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10852.112532] Lustre: Skipped 1 previous similar message [10852.208175] Lustre: lustre-MDT0001: new disk, initializing [10852.360261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10852.467808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10852.492364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10856.640630] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10865.530688] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10870.732985] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10877.355292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10877.533362] Lustre: lustre-OST0000: new disk, initializing [10877.538497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10877.543474] Lustre: 404764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10877.624817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10883.586293] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10885.689879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10885.705981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10885.812122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10892.401131] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10896.498554] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10896.681912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10900.419292] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10900.653408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10906.597545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10906.597649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.599747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10906.612619] LustreError: Skipped 1 previous similar message [10906.619546] Lustre: Skipped 1 previous similar message [10908.695635] Lustre: server umount lustre-OST0000 complete [10913.761927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10913.780914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.798956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10913.808102] Lustre: Skipped 1 previous similar message [10916.837257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10916.840560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.221362] Lustre: server umount lustre-MDT0000 complete [10921.812782] LustreError: 402787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597059 with bad export cookie 7113928012869560502 [10921.820591] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10921.840054] LustreError: 402787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10922.357787] Lustre: server umount lustre-MDT0001 complete [10930.103208] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10931.569385] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:31:06 (1788597066) [10937.481376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10937.929137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10942.015751] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10942.951776] LustreError: 407223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.992811] LustreError: 407223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10948.075232] LustreError: 407222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.699332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.918253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10954.652976] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10962.186109] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10969.742655] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10976.078309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10976.412102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10983.673602] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10990.148677] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10991.674940] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10994.076376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11004.385864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11004.396113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11004.411563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11004.419886] Lustre: Skipped 1 previous similar message [11010.365813] Lustre: server umount lustre-OST0000 complete [11015.652484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.655405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.669453] Lustre: Skipped 2 previous similar messages [11015.674988] Lustre: Skipped 2 previous similar messages [11019.819201] Lustre: server umount lustre-MDT0000 complete [11020.770345] LustreError: 408463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.785019] LustreError: 408463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11023.501904] LustreError: 407202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597160 with bad export cookie 7113928012869561384 [11023.515803] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11023.520506] LustreError: 407202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11024.043479] Lustre: server umount lustre-MDT0001 complete [11036.973490] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11040.325570] Key type lgssc unregistered [11040.677650] LNet: 410951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11040.683630] LNetError: 410951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11040.699407] LNet: Removed LNI 192.168.203.150@tcp [11041.579043] Key type .llcrypt unregistered [11041.586498] Key type ._llcrypt unregistered [11065.527323] Key type ._llcrypt registered [11065.529565] Key type .llcrypt registered [11065.617686] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11065.980336] Key type .llcrypt unregistered [11065.987434] Key type ._llcrypt unregistered [11083.406298] Key type ._llcrypt registered [11083.410138] Key type .llcrypt registered [11083.497477] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11084.668706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11084.704851] alg: No test for adler32 (adler32-zlib) [11085.749238] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11085.995397] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11087.657973] Key type lgssc registered [11088.651722] Lustre: Echo OBD driver; http://www.lustre.org/ [11096.167494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11096.194742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.715378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11102.762832] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11108.323404] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.347088] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11112.529815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11112.955260] LustreError: 413157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11112.979761] LustreError: 413157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11112.981367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11118.051915] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11118.100363] LustreError: 413156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11119.802796] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11123.172242] LustreError: 413152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11123.195674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11123.198334] LustreError: 413152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11127.453783] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11128.294282] LustreError: 413152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.323388] LustreError: 413152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11134.369549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.774546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11139.684173] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11148.769676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11148.792171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.820168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11149.843654] Lustre: server umount lustre-OST0000 complete [11154.918858] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11154.926569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.936293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11155.836467] Lustre: server umount lustre-MDT0000 complete [11159.313819] LustreError: 413137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597296 with bad export cookie 8831872879165564076 [11159.321968] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11159.328485] LustreError: 413137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11168.905672] Lustre: server umount lustre-MDT0001 complete [11179.123857] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11181.903693] Key type lgssc unregistered [11182.182972] LNet: 416304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11182.189379] LNetError: 416304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11182.203692] LNet: Removed LNI 192.168.203.150@tcp [11183.230293] Key type .llcrypt unregistered [11183.232612] Key type ._llcrypt unregistered [11205.598685] Key type ._llcrypt registered [11205.601599] Key type .llcrypt registered [11205.679385] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11206.507103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11206.620283] alg: No test for adler32 (adler32-zlib) [11207.641940] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11207.870763] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11209.592195] Key type lgssc registered [11210.533861] Lustre: Echo OBD driver; http://www.lustre.org/ [11217.052247] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11217.071212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11232.544502] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11237.664302] LustreError: 417691:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [11264.352220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11269.612378] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11276.400226] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11283.718473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.385180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11284.523593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11289.147179] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11289.573588] LustreError: 417716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.613071] LustreError: 417716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11294.696050] LustreError: 417717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.525368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.863305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11300.221553] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11306.577100] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11310.535924] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11317.217500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11317.221640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.228033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11319.778580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11319.786879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.465874] Lustre: server umount lustre-OST0000 complete [11327.458796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11327.464766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.472294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11327.478119] Lustre: Skipped 1 previous similar message [11330.021166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11330.025854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.713708] Lustre: server umount lustre-MDT0000 complete [11334.896917] LustreError: 420218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597472 with bad export cookie 17411659105013124690 [11334.905070] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11334.912680] LustreError: 420218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11335.212960] Lustre: server umount lustre-MDT0001 complete [11344.720294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11347.497575] Key type lgssc unregistered [11347.799125] LNet: 421259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11347.807673] LNetError: 421259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11347.821159] LNet: Removed LNI 192.168.203.150@tcp [11348.906442] Key type .llcrypt unregistered [11348.909798] Key type ._llcrypt unregistered [11372.860195] Key type ._llcrypt registered [11372.862637] Key type .llcrypt registered [11372.970509] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11373.982993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11374.278615] alg: No test for adler32 (adler32-zlib) [11375.304416] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11375.596122] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11377.336178] Key type lgssc registered [11378.757731] Lustre: Echo OBD driver; http://www.lustre.org/ [11386.833876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11386.849976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11388.381093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11392.436727] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11397.608444] LustreError: 422675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.622971] LustreError: 422675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11399.338926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.748317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11403.928874] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11410.307042] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11415.242216] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11421.311263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.622692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11428.284574] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11432.903889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11434.751459] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11440.613202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11440.626514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11440.648911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11441.383747] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [11443.170162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11443.172854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11443.181651] Lustre: Skipped 1 previous similar message [11443.208789] Lustre: Skipped 1 previous similar message [11444.414691] Lustre: server umount lustre-MDT0000 complete [11446.459890] LustreError: 422656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.50@tcp arrived at 1788597583 with bad export cookie 16063483560201787110 [11446.480172] LustreError: 422656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11447.911590] LustreError: 422658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597585 with bad export cookie 16063483560201786214 [11447.921711] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11448.290466] LustreError: 422675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.292070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.306868] LustreError: 422675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11448.340246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11448.345061] Lustre: Skipped 1 previous similar message [11452.323486] LustreError: 422670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.337645] LustreError: 422670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11454.151845] Lustre: server umount lustre-OST0000 complete [11457.505480] LustreError: 422675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.521101] LustreError: 422675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11461.150223] Lustre: server umount lustre-MDT0001 complete [11472.094936] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11475.549859] Key type lgssc unregistered [11475.818899] LNet: 426357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11475.830823] LNetError: 426357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11476.921478] LNet: Removed LNI 192.168.203.150@tcp [11477.782529] Key type .llcrypt unregistered [11477.788226] Key type ._llcrypt unregistered [11501.270810] Key type ._llcrypt registered [11501.274486] Key type .llcrypt registered [11501.437787] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11502.367155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11502.382700] alg: No test for adler32 (adler32-zlib) [11503.845851] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11504.275687] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11506.056162] Key type lgssc registered [11507.793583] Lustre: Echo OBD driver; http://www.lustre.org/ [11516.993409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11517.016222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.481798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11523.140441] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11527.662468] LustreError: 427769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11527.679848] LustreError: 427769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11530.615837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.981564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11535.471590] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11541.850840] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11546.721325] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11552.785509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11553.319968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11559.669789] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11563.577460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11568.492139] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11573.737643] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11573.738719] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11573.744825] Lustre: Skipped 1 previous similar message [11573.762375] Lustre: Skipped 1 previous similar message [11575.522369] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [11578.835986] Lustre: server umount lustre-MDT0001 complete [11578.860609] LustreError: 427768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.887559] LustreError: 427768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11583.971859] LustreError: 427769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.089844] LustreError: 428497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.092079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11589.112377] LustreError: 428497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11589.126145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11590.898886] Lustre: server umount lustre-OST0000 complete [11594.399201] Lustre: server umount lustre-MDT0000 complete [11606.442846] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11609.263559] Key type lgssc unregistered [11609.540661] LNet: 431452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11609.553639] LNetError: 431452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11609.584824] LNet: Removed LNI 192.168.203.150@tcp [11610.444321] Key type .llcrypt unregistered [11610.446559] Key type ._llcrypt unregistered [11634.555900] Key type ._llcrypt registered [11634.558662] Key type .llcrypt registered [11634.659669] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [11648.486439] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11649.331318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11649.358203] alg: No test for adler32 (adler32-zlib) [11650.432353] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11650.666403] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11652.384191] Key type lgssc registered [11653.383055] Lustre: Echo OBD driver; http://www.lustre.org/ [11693.198734] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11710.193533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11710.252361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11711.760774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11711.875444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11712.019537] Lustre: lustre-MDT0000: new disk, initializing [11712.153259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11712.176132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11716.490730] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11732.001139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11732.220642] Lustre: lustre-OST0000: new disk, initializing [11732.223943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11732.227705] Lustre: Skipped 1 previous similar message [11732.231858] Lustre: 436474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11732.292995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11737.760114] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11740.215146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11740.219916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11740.242680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11746.780380] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11766.253320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11766.527586] Lustre: 435456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11766.663801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11766.781294] Lustre: lustre-MDT0001: new disk, initializing [11767.018342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11767.103932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11767.114524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11773.176299] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11784.712402] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11785.027862] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11790.234550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11790.387126] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11795.697575] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [11799.521990] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.526105] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11799.573969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11799.600946] Lustre: Skipped 1 previous similar message [11800.782125] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [11802.089937] Lustre: server umount lustre-MDT0001 complete [11809.784638] LustreError: 435463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.821048] LustreError: 435463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11814.890226] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11814.909613] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11820.011401] LustreError: 436482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.044529] LustreError: 436482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11822.049506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11822.064138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.089283] Lustre: Skipped 1 previous similar message [11822.098663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11825.125551] LustreError: 435468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.241443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11830.242501] LustreError: 435467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.251087] Lustre: Skipped 1 previous similar message [11835.360142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11835.722946] Lustre: server umount lustre-OST0000 complete [11846.101617] Lustre: server umount lustre-MDT0000 complete [11859.334840] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11861.916104] Key type lgssc unregistered [11862.157609] LNet: 440220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11862.167339] LNetError: 440220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11862.181638] LNet: Removed LNI 192.168.203.150@tcp [11863.002175] Key type .llcrypt unregistered [11863.006363] Key type ._llcrypt unregistered [11905.478996] Key type ._llcrypt registered [11905.481965] Key type .llcrypt registered [11905.652788] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11906.001361] Key type .llcrypt unregistered [11906.003219] Key type ._llcrypt unregistered [11921.171204] Key type ._llcrypt registered [11921.173516] Key type .llcrypt registered [11921.267989] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11922.436630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11922.543123] alg: No test for adler32 (adler32-zlib) [11923.585766] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11923.784607] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11925.419670] Key type lgssc registered [11926.646788] Lustre: Echo OBD driver; http://www.lustre.org/ [11954.683303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11954.718686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.938753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11955.969385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11956.070355] Lustre: lustre-MDT0000: new disk, initializing [11956.230693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11956.263920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11960.437421] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11970.404395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.533066] Lustre: 443601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11970.617175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11970.620255] Lustre: Skipped 1 previous similar message [11970.683855] Lustre: lustre-MDT0001: new disk, initializing [11970.768870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11970.811768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11970.837214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11975.003184] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11984.176210] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11991.379430] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11999.673660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12000.155674] Lustre: lustre-OST0000: new disk, initializing [12000.168525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12000.179638] Lustre: 445571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12000.347307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12006.918370] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12010.582661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12010.594604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12010.706284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12016.231890] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12022.241732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12022.259236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.279286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12026.343937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12026.356239] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12028.128115] Lustre: server umount lustre-MDT0000 complete [12031.710428] LustreError: 443594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598169 with bad export cookie 2597205842427450597 [12031.722246] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12031.728348] LustreError: 443594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12031.959940] Lustre: server umount lustre-MDT0001 complete [12044.880657] Lustre: server umount lustre-OST0000 complete [12055.022807] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:49:50 (1788598190) [12061.383565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.778685] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.803358] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12061.879922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12065.933406] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12067.298694] LustreError: 447599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.430078] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.243358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.638134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12078.180327] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12085.665389] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12090.776350] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12110.185846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.408063] Lustre: Found index 0 for lustre-OST0000, updating log [12110.625514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.584677] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12124.779813] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12133.240880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12143.586785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.590094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12143.593671] Lustre: Skipped 1 previous similar message [12143.595680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12143.624825] Lustre: Skipped 2 previous similar messages [12147.642170] Lustre: server umount lustre-OST0000 complete [12165.601583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12165.616784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.627291] Lustre: Skipped 1 previous similar message [12165.637263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.642072] Lustre: Skipped 3 previous similar messages [12170.865855] Lustre: server umount lustre-MDT0000 complete [12171.247874] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.279294] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12173.673960] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [12175.570527] LustreError: 447578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598312 with bad export cookie 2597205842427451990 [12175.580436] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12175.581969] LustreError: 447578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12175.906837] Lustre: server umount lustre-MDT0001 complete [12186.856851] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:52:01 (1788598321) [12807.362122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12807.990890] LustreError: 454038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12808.023988] LustreError: 454038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12808.093143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12812.410652] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12813.281769] LustreError: 454039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.338446] LustreError: 454039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12818.418199] LustreError: 454038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.140508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.707868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12827.954576] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12836.037348] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12841.620766] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12862.898279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12863.188954] Lustre: Found index 0 for lustre-OST0000, updating log [12863.566417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12864.930594] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [12870.128604] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12878.550933] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12880.691171] Lustre: server umount lustre-OST0000 complete [12885.667635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12885.682960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12885.701080] Lustre: Skipped 2 previous similar messages [12885.718671] Lustre: Skipped 3 previous similar messages [12886.754255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12886.764897] Lustre: Skipped 1 previous similar message [12887.779758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12887.784962] Lustre: Skipped 1 previous similar message [12889.827916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12889.844032] Lustre: Skipped 3 previous similar messages [12891.060601] Lustre: server umount lustre-MDT0000 complete [12891.876691] LustreError: 454767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12891.901941] LustreError: 454767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12894.919423] LustreError: 454021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599032 with bad export cookie 2597205842427453068 [12894.930407] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12894.940416] LustreError: 454021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12895.455954] Lustre: server umount lustre-MDT0001 complete [12925.344873] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [12927.651328] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [12938.411616] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12979.131152] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:05:14 (1788599114) [12997.947702] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [13000.803371] Key type lgssc unregistered [13001.141981] LNet: 460651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13001.157235] LNetError: 460651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13125.094818] LNet: Removed LNI 192.168.203.150@tcp [13125.990393] Key type .llcrypt unregistered [13125.999109] Key type ._llcrypt unregistered [13143.838942] Key type ._llcrypt registered [13143.841341] Key type .llcrypt registered [13143.993843] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13144.888264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13144.930332] alg: No test for adler32 (adler32-zlib) [13146.003567] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [13146.291575] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [13148.012085] Key type lgssc registered [13149.335927] Lustre: Echo OBD driver; http://www.lustre.org/ [13153.343584] LNet: Added LNI 192.168.203.150@tcp100 [8/256/0/180] [13154.085603] LNet: Added LNI 192.168.203.150@tcp101 [8/256/0/180] [13154.798505] LNet: Added LNI 192.168.203.150@tcp102 [8/256/0/180] [13155.666120] LNet: Added LNI 192.168.203.150@tcp103 [8/256/0/180] [13156.577111] LNet: Added LNI 192.168.203.150@tcp104 [8/256/0/180] [13157.433888] LNet: Added LNI 192.168.203.150@tcp105 [8/256/0/180] [13158.294276] LNet: Added LNI 192.168.203.150@tcp106 [8/256/0/180] [13159.128658] LNet: Added LNI 192.168.203.150@tcp107 [8/256/0/180] [13159.918437] LNet: Added LNI 192.168.203.150@tcp108 [8/256/0/180] [13160.801496] LNet: Added LNI 192.168.203.150@tcp109 [8/256/0/180] [13161.644965] LNet: Added LNI 192.168.203.150@tcp110 [8/256/0/180] [13162.483508] LNet: Added LNI 192.168.203.150@tcp111 [8/256/0/180] [13163.335614] LNet: Added LNI 192.168.203.150@tcp112 [8/256/0/180] [13164.156331] LNet: Added LNI 192.168.203.150@tcp113 [8/256/0/180] [13164.910441] LNet: Added LNI 192.168.203.150@tcp114 [8/256/0/180] [13165.734691] LNet: Added LNI 192.168.203.150@tcp115 [8/256/0/180] [13166.626015] LNet: Added LNI 192.168.203.150@tcp116 [8/256/0/180] [13167.566130] LNet: Added LNI 192.168.203.150@tcp117 [8/256/0/180] [13168.306580] LNet: Added LNI 192.168.203.150@tcp118 [8/256/0/180] [13169.073844] LNet: Added LNI 192.168.203.150@tcp119 [8/256/0/180] [13169.919870] LNet: Added LNI 192.168.203.150@tcp120 [8/256/0/180] [13170.619029] LNet: Added LNI 192.168.203.150@tcp121 [8/256/0/180] [13171.300451] LNet: Added LNI 192.168.203.150@tcp122 [8/256/0/180] [13172.157755] LNet: Added LNI 192.168.203.150@tcp123 [8/256/0/180] [13172.973610] LNet: Added LNI 192.168.203.150@tcp124 [8/256/0/180] [13173.720769] LNet: Added LNI 192.168.203.150@tcp125 [8/256/0/180] [13174.641186] LNet: Added LNI 192.168.203.150@tcp126 [8/256/0/180] [13175.396936] LNet: Added LNI 192.168.203.150@tcp127 [8/256/0/180] [13176.391339] LNet: Added LNI 192.168.203.150@tcp128 [8/256/0/180] [13177.237471] LNet: Added LNI 192.168.203.150@tcp129 [8/256/0/180] [13178.060354] LNet: Added LNI 192.168.203.150@tcp130 [8/256/0/180] [13178.953774] LNet: Added LNI 192.168.203.150@tcp131 [8/256/0/180] [13179.789976] LNet: Added LNI 192.168.203.150@tcp132 [8/256/0/180] [13180.580692] LNet: Added LNI 192.168.203.150@tcp133 [8/256/0/180] [13181.440632] LNet: Added LNI 192.168.203.150@tcp134 [8/256/0/180] [13182.489975] LNet: Added LNI 192.168.203.150@tcp135 [8/256/0/180] [13189.613038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13189.636411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.797566] Lustre: 463680:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [13190.952963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13190.979885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13191.089116] Lustre: lustre-MDT0000: new disk, initializing [13191.268761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13191.304058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13196.113770] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13207.040064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13207.243930] Lustre: 463695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13207.364816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13207.374617] Lustre: Skipped 1 previous similar message [13207.511678] Lustre: lustre-MDT0001: new disk, initializing [13207.564901] Lustre: 464495:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [13207.583964] Lustre: 464495:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13207.769319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13207.832815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13207.859177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13209.790946] Lustre: 464577:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [13209.827842] Lustre: 464577:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13213.519430] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13224.500672] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13230.622640] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13236.965611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13237.381228] Lustre: lustre-OST0000: new disk, initializing [13237.387984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13237.398137] Lustre: 465669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13237.479196] Lustre: 465674:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [13237.492419] Lustre: 465674:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13237.526176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13244.302240] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13247.062229] Lustre: 466126:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [13247.069364] Lustre: 466126:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13247.172541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13247.181120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13247.315441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13254.089266] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13262.823509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13262.824337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.854318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13262.867613] Lustre: Skipped 1 previous similar message [13262.894236] Lustre: Skipped 1 previous similar message [13267.952721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13267.963248] Lustre: Skipped 1 previous similar message [13268.784601] Lustre: server umount lustre-OST0000 complete [13274.082870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13274.100329] LustreError: Skipped 1 previous similar message [13274.102498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13274.142733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13278.179744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13278.183770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13279.717244] Lustre: server umount lustre-MDT0000 complete [13283.862242] LustreError: 463688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599421 with bad export cookie 8321434337303372687 [13283.869496] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13283.873545] LustreError: 463688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13284.232107] Lustre: server umount lustre-MDT0001 complete [13295.436705] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [13298.528726] Key type lgssc unregistered [13298.919542] LNet: 467583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13298.929313] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13298.936320] LNet: Removed LNI 192.168.203.150@tcp [13300.000635] LNet: Removed LNI 192.168.203.150@tcp100 [13300.005845] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13300.017533] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13300.029496] LNet: Removed LNI 192.168.203.150@tcp101 [13301.088201] LNet: Removed LNI 192.168.203.150@tcp102 [13301.090346] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13301.098306] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13301.118402] LNet: Removed LNI 192.168.203.150@tcp103 [13302.176180] LNet: Removed LNI 192.168.203.150@tcp104 [13303.200283] LNet: Removed LNI 192.168.203.150@tcp105 [13303.203791] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13303.209642] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13303.218562] LNet: Removed LNI 192.168.203.150@tcp106 [13304.224163] LNet: Removed LNI 192.168.203.150@tcp107 [13305.248289] LNet: Removed LNI 192.168.203.150@tcp108 [13306.272740] LNet: Removed LNI 192.168.203.150@tcp109 [13307.296142] LNet: Removed LNI 192.168.203.150@tcp110 [13307.311780] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13307.318344] LNetError: 467583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13308.384891] LNet: Removed LNI 192.168.203.150@tcp111 [13309.409977] LNet: Removed LNI 192.168.203.150@tcp112 [13309.417741] LNet: Removed LNI 192.168.203.150@tcp113 [13309.429624] LNet: Removed LNI 192.168.203.150@tcp114 [13309.435510] LNet: Removed LNI 192.168.203.150@tcp115 [13309.446082] LNet: Removed LNI 192.168.203.150@tcp116 [13309.454526] LNet: Removed LNI 192.168.203.150@tcp117 [13309.462165] LNet: Removed LNI 192.168.203.150@tcp118 [13309.469462] LNet: Removed LNI 192.168.203.150@tcp119 [13309.474185] LNet: Removed LNI 192.168.203.150@tcp120 [13309.480458] LNet: Removed LNI 192.168.203.150@tcp121 [13309.483236] LNet: Removed LNI 192.168.203.150@tcp122 [13309.486309] LNet: Removed LNI 192.168.203.150@tcp123 [13309.493820] LNet: Removed LNI 192.168.203.150@tcp124 [13309.499809] LNet: Removed LNI 192.168.203.150@tcp125 [13309.509196] LNet: Removed LNI 192.168.203.150@tcp126 [13309.513187] LNet: Removed LNI 192.168.203.150@tcp127 [13309.516580] LNet: Removed LNI 192.168.203.150@tcp128 [13309.524623] LNet: Removed LNI 192.168.203.150@tcp129 [13309.530651] LNet: Removed LNI 192.168.203.150@tcp130 [13309.540913] LNet: Removed LNI 192.168.203.150@tcp131 [13309.565776] LNet: Removed LNI 192.168.203.150@tcp132 [13309.571585] LNet: Removed LNI 192.168.203.150@tcp133 [13309.575574] LNet: Removed LNI 192.168.203.150@tcp134 [13309.590204] LNet: Removed LNI 192.168.203.150@tcp135 [13310.545661] Key type .llcrypt unregistered [13310.551540] Key type ._llcrypt unregistered [13336.166364] Key type ._llcrypt registered [13336.171047] Key type .llcrypt registered [13336.304282] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [13351.540668] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13352.641656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13352.732135] alg: No test for adler32 (adler32-zlib) [13353.739846] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [13354.040597] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [13355.736220] Key type lgssc registered [13356.805861] Lustre: Echo OBD driver; http://www.lustre.org/ [13392.323723] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:12:07 (1788599527) [13413.074348] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [13416.271456] Key type lgssc unregistered [13416.525418] LNet: 471666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13416.542542] LNetError: 471666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13416.565910] LNet: Removed LNI 192.168.203.150@tcp [13417.598699] Key type .llcrypt unregistered [13417.600252] Key type ._llcrypt unregistered [13434.533372] Key type ._llcrypt registered [13434.537332] Key type .llcrypt registered [13434.651684] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13435.934861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13435.969871] alg: No test for adler32 (adler32-zlib) [13437.040093] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [13437.253329] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [13438.912438] Key type lgssc registered [13439.981872] Lustre: Echo OBD driver; http://www.lustre.org/ [13448.976940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13449.007598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13450.231876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13450.260884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13450.326487] Lustre: lustre-MDT0000: new disk, initializing [13450.383462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13450.403663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13455.419545] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13466.295228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13466.424262] Lustre: 472875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13466.480604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13466.488762] Lustre: Skipped 1 previous similar message [13466.601257] Lustre: lustre-MDT0001: new disk, initializing [13466.712081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13466.782080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13466.795557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13470.930511] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13480.581695] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13486.202457] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13493.488121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13493.875488] Lustre: lustre-OST0000: new disk, initializing [13493.880778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13493.902752] Lustre: 474844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13494.047418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13500.477247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13500.504225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13500.602677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13501.220696] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13511.202778] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13515.566508] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing lnet_if_list [13521.100985] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13521.127226] LNet: Added LNI 192.168.203.150@tcp10 [8/256/0/180] [13528.306563] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13528.378801] LNet: Added LNI 192.168.203.150@tcp11 [8/256/0/180] [13533.337603] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13533.405365] LNet: Added LNI 192.168.203.150@tcp12 [8/256/0/180] [13538.379698] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13538.408556] LNet: Added LNI 192.168.203.150@tcp13 [8/256/0/180] [13544.397264] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13544.446590] LNet: Added LNI 192.168.203.150@tcp14 [8/256/0/180] [13550.729454] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13550.769040] LNet: Added LNI 192.168.203.150@tcp15 [8/256/0/180] [13556.378935] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13556.424382] LNet: Added LNI 192.168.203.150@tcp16 [8/256/0/180] [13561.727931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13561.760315] LNet: Added LNI 192.168.203.150@tcp17 [8/256/0/180] [13567.782678] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13567.827739] LNet: Added LNI 192.168.203.150@tcp18 [8/256/0/180] [13573.140861] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13573.163487] LNet: Added LNI 192.168.203.150@tcp19 [8/256/0/180] [13579.475703] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13579.497875] LNet: Added LNI 192.168.203.150@tcp20 [8/256/0/180] [13585.441781] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13585.474893] LNet: Added LNI 192.168.203.150@tcp21 [8/256/0/180] [13591.267470] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13591.334424] LNet: Added LNI 192.168.203.150@tcp22 [8/256/0/180] [13597.211803] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13597.241680] LNet: Added LNI 192.168.203.150@tcp23 [8/256/0/180] [13603.524293] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13603.569897] LNet: Added LNI 192.168.203.150@tcp24 [8/256/0/180] [13609.146947] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13609.195196] LNet: Added LNI 192.168.203.150@tcp25 [8/256/0/180] [13614.660975] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13614.698929] LNet: Added LNI 192.168.203.150@tcp26 [8/256/0/180] [13621.106850] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13621.144812] LNet: Added LNI 192.168.203.150@tcp27 [8/256/0/180] [13627.972994] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13627.998601] LNet: Added LNI 192.168.203.150@tcp28 [8/256/0/180] [13633.635457] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13633.661895] LNet: Added LNI 192.168.203.150@tcp29 [8/256/0/180] [13639.081723] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13639.129300] LNet: Added LNI 192.168.203.150@tcp30 [8/256/0/180] [13644.936051] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13644.974639] LNet: Added LNI 192.168.203.150@tcp31 [8/256/0/180] [13651.232837] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13651.279585] LNet: Added LNI 192.168.203.150@tcp32 [8/256/0/180] [13656.943381] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13656.966335] LNet: Added LNI 192.168.203.150@tcp33 [8/256/0/180] [13662.536148] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13662.557580] LNet: Added LNI 192.168.203.150@tcp34 [8/256/0/180] [13668.599547] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13668.626844] LNet: Added LNI 192.168.203.150@tcp35 [8/256/0/180] [13673.829697] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13673.855473] LNet: Added LNI 192.168.203.150@tcp36 [8/256/0/180] [13679.442930] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13679.463991] LNet: Added LNI 192.168.203.150@tcp37 [8/256/0/180] [13686.261038] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13686.300948] LNet: Added LNI 192.168.203.150@tcp38 [8/256/0/180] [13694.598769] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13694.639019] LNet: Added LNI 192.168.203.150@tcp39 [8/256/0/180] [13703.505642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13703.535086] LNet: Added LNI 192.168.203.150@tcp40 [8/256/0/180] [13711.128966] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13711.169452] LNet: Added LNI 192.168.203.150@tcp41 [8/256/0/180] [13717.605131] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13717.649531] LNet: Added LNI 192.168.203.150@tcp42 [8/256/0/180] [13725.158165] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13725.197656] LNet: Added LNI 192.168.203.150@tcp43 [8/256/0/180] [13733.996612] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13734.035646] LNet: Added LNI 192.168.203.150@tcp44 [8/256/0/180] [13743.233923] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13743.265312] LNet: Added LNI 192.168.203.150@tcp45 [8/256/0/180] [13751.723434] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13751.758414] LNet: Added LNI 192.168.203.150@tcp46 [8/256/0/180] [13759.604236] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13759.628476] LNet: Added LNI 192.168.203.150@tcp47 [8/256/0/180] [13767.781122] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13767.814438] LNet: Added LNI 192.168.203.150@tcp48 [8/256/0/180] [13776.398705] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13776.449542] LNet: Added LNI 192.168.203.150@tcp49 [8/256/0/180] [13786.159177] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13786.188510] LNet: Added LNI 192.168.203.150@tcp50 [8/256/0/180] [13808.099829] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13808.104830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13808.116811] Lustre: Skipped 1 previous similar message [13808.138728] Lustre: Skipped 2 previous similar messages [13811.694579] Lustre: server umount lustre-MDT0000 complete [13815.743437] LustreError: 472866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599953 with bad export cookie 15282848314562040390 [13815.746349] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13815.755379] LustreError: 472866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13816.190799] Lustre: server umount lustre-MDT0001 complete [13833.697220] Lustre: 472148:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788599955/real 1788599955] req@000000006645b61e x1875482587081344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788599971 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13833.717660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13833.726539] Lustre: Skipped 1 previous similar message [13835.048528] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13835.744078] Lustre: 472146:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788599957/real 1788599957] req@00000000c8bb331b x1875482587081472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788599973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13838.945151] Lustre: 472146:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788599960/real 1788599960] req@00000000531360d2 x1875482587081600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788599976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13841.888184] Lustre: 472148:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788599963/real 1788599963] req@0000000083244826 x1875482587081856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788599979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13844.616024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13848.096396] Lustre: 472148:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788599969/real 1788599969] req@00000000d67aa0da x1875482587081984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788599985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13861.688518] LustreError: 474837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.711107] LustreError: 474837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13861.749282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13862.755859] LustreError: 474838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13862.789960] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13864.804719] LustreError: 474837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13866.245358] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13870.072931] LustreError: 474838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13875.173595] LustreError: 474837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13875.532840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13875.902918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13878.977658] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13880.688353] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13888.785292] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13894.551710] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13912.037848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13912.045938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13912.063443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13913.058593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13913.073859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13915.108307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13915.117212] Lustre: Skipped 1 previous similar message [13917.794752] Lustre: server umount lustre-OST0000 complete [13922.273053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13922.283261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13922.300315] Lustre: Skipped 1 previous similar message [13922.306068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13927.389111] Lustre: server umount lustre-MDT0000 complete [13930.465058] LustreError: 499091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13930.484764] LustreError: 499091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13930.882426] LustreError: 472866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600068 with bad export cookie 15282848314562043043 [13930.888560] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13930.899579] LustreError: 472866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13931.206470] Lustre: server umount lustre-MDT0001 complete [13942.285658] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [13944.897449] Key type lgssc unregistered [13945.260097] LNet: 501865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13945.265123] LNetError: 501865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13945.269153] LNet: Removed LNI 192.168.203.150@tcp [13945.273925] LNet: Removed LNI 192.168.203.150@tcp10 [13945.276733] LNet: Removed LNI 192.168.203.150@tcp11 [13945.279567] LNet: Removed LNI 192.168.203.150@tcp12 [13945.283695] LNet: Removed LNI 192.168.203.150@tcp13 [13945.287186] LNet: Removed LNI 192.168.203.150@tcp14 [13945.291665] LNet: Removed LNI 192.168.203.150@tcp15 [13945.295947] LNet: Removed LNI 192.168.203.150@tcp16 [13945.299379] LNet: Removed LNI 192.168.203.150@tcp17 [13945.301216] LNet: Removed LNI 192.168.203.150@tcp18 [13945.303361] LNet: Removed LNI 192.168.203.150@tcp19 [13945.306323] LNet: Removed LNI 192.168.203.150@tcp20 [13945.308275] LNet: Removed LNI 192.168.203.150@tcp21 [13945.311545] LNet: Removed LNI 192.168.203.150@tcp22 [13945.316361] LNet: Removed LNI 192.168.203.150@tcp23 [13945.318803] LNet: Removed LNI 192.168.203.150@tcp24 [13945.322677] LNet: Removed LNI 192.168.203.150@tcp25 [13945.324541] LNet: Removed LNI 192.168.203.150@tcp26 [13945.337202] LNet: Removed LNI 192.168.203.150@tcp27 [13945.345220] LNet: Removed LNI 192.168.203.150@tcp28 [13945.347199] LNet: Removed LNI 192.168.203.150@tcp29 [13945.352216] LNet: Removed LNI 192.168.203.150@tcp30 [13945.355017] LNet: Removed LNI 192.168.203.150@tcp31 [13945.357957] LNet: Removed LNI 192.168.203.150@tcp32 [13945.360645] LNet: Removed LNI 192.168.203.150@tcp33 [13945.363435] LNet: Removed LNI 192.168.203.150@tcp34 [13945.366144] LNet: Removed LNI 192.168.203.150@tcp35 [13945.375291] LNet: Removed LNI 192.168.203.150@tcp36 [13945.383723] LNet: Removed LNI 192.168.203.150@tcp37 [13945.388995] LNet: Removed LNI 192.168.203.150@tcp38 [13945.394380] LNet: Removed LNI 192.168.203.150@tcp39 [13945.396834] LNet: Removed LNI 192.168.203.150@tcp40 [13945.400424] LNet: Removed LNI 192.168.203.150@tcp41 [13945.402660] LNet: Removed LNI 192.168.203.150@tcp42 [13945.405892] LNet: Removed LNI 192.168.203.150@tcp43 [13945.407785] LNet: Removed LNI 192.168.203.150@tcp44 [13945.414683] LNet: Removed LNI 192.168.203.150@tcp45 [13945.417223] LNet: Removed LNI 192.168.203.150@tcp46 [13945.421755] LNet: Removed LNI 192.168.203.150@tcp47 [13945.424465] LNet: Removed LNI 192.168.203.150@tcp48 [13945.426741] LNet: Removed LNI 192.168.203.150@tcp49 [13945.437721] LNet: Removed LNI 192.168.203.150@tcp50 [13946.178189] Key type .llcrypt unregistered [13946.180213] Key type ._llcrypt unregistered [13963.423436] Key type ._llcrypt registered [13963.426550] Key type .llcrypt registered [13963.526404] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [13976.526170] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13977.522890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13977.667911] alg: No test for adler32 (adler32-zlib) [13978.842584] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [13979.104581] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [13980.794606] Key type lgssc registered [13982.273892] Lustre: Echo OBD driver; http://www.lustre.org/ [14015.843185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14015.857535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14017.104295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14017.154276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14017.223234] Lustre: lustre-MDT0000: new disk, initializing [14017.365727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14017.451977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14021.799995] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14031.395089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14031.516350] Lustre: 505038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14031.591686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14031.609153] Lustre: Skipped 1 previous similar message [14031.724735] Lustre: lustre-MDT0001: new disk, initializing [14031.824378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14031.874064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14031.886843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14036.168266] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14044.971862] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14050.163645] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14056.921929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14057.111803] Lustre: *** cfs_fail_loc=90f, val=0*** [14057.162951] Lustre: lustre-OST0000: new disk, initializing [14057.166189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14057.170888] Lustre: 507052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14057.239843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14062.942837] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14066.270920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14066.288225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14066.420271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14071.589731] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14082.024801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14082.026221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14082.032374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14082.041751] Lustre: Skipped 1 previous similar message [14082.067372] LustreError: Skipped 1 previous similar message [14084.605666] Lustre: server umount lustre-OST0000 complete [14088.161689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14088.173400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14088.196890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14088.202894] Lustre: Skipped 1 previous similar message [14092.258267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14092.269448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14094.404409] Lustre: server umount lustre-MDT0000 complete [14098.049164] LustreError: 505031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600235 with bad export cookie 17004266096873479455 [14098.067330] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14098.455896] Lustre: server umount lustre-MDT0001 complete [14110.122706] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [14113.369433] Key type lgssc unregistered [14113.665211] LNet: 508956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14113.677672] LNetError: 508956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14113.705916] LNet: Removed LNI 192.168.203.150@tcp [14114.753766] Key type .llcrypt unregistered [14114.756354] Key type ._llcrypt unregistered [14135.609965] Key type ._llcrypt registered [14135.614450] Key type .llcrypt registered [14135.721730] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14136.955148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14137.062087] alg: No test for adler32 (adler32-zlib) [14138.225995] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [14138.545816] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [14140.288180] Key type lgssc registered [14141.849142] Lustre: Echo OBD driver; http://www.lustre.org/ [14149.469793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14149.502165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14151.052600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14155.595093] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14161.383862] LustreError: 510222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14161.405771] LustreError: 510222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14163.104292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14163.375185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14167.508181] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14174.247945] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14178.914781] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14184.614701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14184.902541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14190.097199] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14196.949273] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14206.830260] loop0: detected capacity change from 0 to 204800000 [14206.995565] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14207.006332] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14207.015697] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14212.076393] loop0: detected capacity change from 0 to 204800000 [14212.227047] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14212.239479] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14212.255725] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14217.726840] loop0: detected capacity change from 0 to 204800000 [14223.846837] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14224.011202] Lustre: fs15246-OST0000: new disk, initializing [14224.014201] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14224.018524] Lustre: 513814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14224.054988] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14230.263980] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14238.617844] loop1: detected capacity change from 0 to 204800000 [14242.637810] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14242.716748] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14242.770971] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14242.821699] Lustre: fs15246-MDT0000: new disk, initializing [14242.928836] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14242.944715] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14243.030448] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14244.068351] Lustre: 514939:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000b9a67f2 x1875483322001920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788600397 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14246.603817] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14253.220982] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14253.228849] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14253.238870] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14266.867431] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14266.893694] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14270.911537] Lustre: server umount fs15246-MDT0000 complete [14271.974161] LustreError: 512063:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14277.098284] LustreError: 510219:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14278.973766] Lustre: server umount fs15246-OST0000 complete [14288.361093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14288.371880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14288.389960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14290.401483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14290.407520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14290.415827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14292.451919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14292.462104] Lustre: Skipped 1 previous similar message [14294.377332] Lustre: server umount lustre-OST0000 complete [14301.666412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14301.673682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14301.686696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14305.204509] Lustre: server umount lustre-MDT0000 complete [14307.814113] LustreError: 512063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14307.832592] LustreError: 512063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14308.887060] LustreError: 510203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600446 with bad export cookie 2502732303661089209 [14308.897504] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14308.899418] LustreError: 510203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14309.232865] Lustre: server umount lustre-MDT0001 complete [14319.318428] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [14322.119566] Key type lgssc unregistered [14322.466630] LNet: 518033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14322.481949] LNetError: 518033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14322.501305] LNet: Removed LNI 192.168.203.150@tcp [14323.325854] Key type .llcrypt unregistered [14323.327702] Key type ._llcrypt unregistered [14339.075895] Key type ._llcrypt registered [14339.078182] Key type .llcrypt registered [14339.171368] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14340.230846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14340.267166] alg: No test for adler32 (adler32-zlib) [14341.291571] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [14341.522689] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [14343.187574] Key type lgssc registered [14344.524950] Lustre: Echo OBD driver; http://www.lustre.org/ [14379.847472] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [14389.535594] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14417.834558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14417.875699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14419.087798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14419.115749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14419.202524] Lustre: lustre-MDT0000: new disk, initializing [14419.317829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14419.334869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14423.330546] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14432.590412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14432.718428] Lustre: 522495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14432.774587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14432.777879] Lustre: Skipped 1 previous similar message [14432.836911] Lustre: lustre-MDT0001: new disk, initializing [14432.958038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14432.995564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14433.011839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14436.978869] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14444.406288] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14449.495286] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14455.857037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14456.084897] Lustre: lustre-OST0000: new disk, initializing [14456.092687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14456.105066] Lustre: 524462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14456.206496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14462.377128] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14463.052601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14463.058739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14463.196759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14470.449412] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14474.663410] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14474.860648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14478.423904] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14478.619603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14483.938526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14483.945480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14483.952572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14483.964128] Lustre: Skipped 1 previous similar message [14483.982647] Lustre: Skipped 1 previous similar message [14486.457601] Lustre: server umount lustre-OST0000 complete [14494.186986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14494.206704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14494.209566] Lustre: Skipped 1 previous similar message [14494.246652] Lustre: Skipped 1 previous similar message [14496.282599] Lustre: server umount lustre-MDT0000 complete [14499.789616] LustreError: 522486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600637 with bad export cookie 11143708142239456890 [14499.793102] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14499.799798] LustreError: 522486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14500.204747] Lustre: server umount lustre-MDT0001 complete [14526.674455] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [14536.169549] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14564.144852] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14262 sec ========= 05:31:39 (1788600699) [14565.565287] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:31:41 (1788600701) === [14568.209481] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:31:43 (1788600703) === [14600.923725] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [14603.202449] Key type lgssc unregistered [14603.497889] LNet: 531105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14603.511734] LNetError: 531105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14603.532557] LNet: Removed LNI 192.168.203.150@tcp [14604.398276] Key type .llcrypt unregistered [14604.400889] Key type ._llcrypt unregistered