[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 446658934 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002361] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010090] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013078] *** VALIDATE selinux *** [ 0.022004] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027423] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031021] *** VALIDATE tmpfs *** [ 0.032468] *** VALIDATE proc *** [ 0.033275] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036028] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043080] debug: unmapping init [mem 0xffffffffb1e59000-0xffffffffb1e60fff] [ 0.045928] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046755] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048021] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054337] rcu: Hierarchical SRCU implementation. [ 0.056491] smp: Bringing up secondary CPUs ... [ 0.057572] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.061449] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136406] node 0 deferred pages initialised in 69ms [ 0.139107] devtmpfs: initialized [ 0.140241] x86/mm: Memory block size: 128MB [ 0.143048] gcov: version magic: 0x41383552 [ 0.147041] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149289] pinctrl core: initialized pinctrl subsystem [ 0.150175] [ 0.150712] ************************************************************* [ 0.151013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152014] ** ** [ 0.153014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154012] ** ** [ 0.155018] ** This means that this kernel is built to expose internal ** [ 0.156016] ** IOMMU data structures, which may compromise security on ** [ 0.157014] ** your system. ** [ 0.158017] ** ** [ 0.159014] ** If you see this message and you are not debugging the ** [ 0.160013] ** kernel, report this immediately to your vendor! ** [ 0.161013] ** ** [ 0.162015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163020] ************************************************************* [ 0.164788] NET: Registered protocol family 16 [ 0.165486] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169137] cpuidle: using governor menu [ 0.171917] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174466] PCI: Using configuration type 1 for base access [ 0.176137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186176] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193216] cryptd: max_cpu_qlen set to 1000 [ 0.196281] ACPI: Added _OSI(Module Device) [ 0.198017] ACPI: Added _OSI(Processor Device) [ 0.200015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207859] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215061] ACPI: Interpreter enabled [ 0.216077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217014] ACPI: Using IOAPIC for interrupt routing [ 0.218135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221502] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233153] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245486] acpiphp: Slot [2] registered [ 0.247107] acpiphp: Slot [5] registered [ 0.248121] acpiphp: Slot [6] registered [ 0.249115] acpiphp: Slot [7] registered [ 0.251133] acpiphp: Slot [8] registered [ 0.253114] acpiphp: Slot [9] registered [ 0.254196] acpiphp: Slot [10] registered [ 0.256134] acpiphp: Slot [3] registered [ 0.258150] acpiphp: Slot [4] registered [ 0.259083] acpiphp: Slot [11] registered [ 0.261108] acpiphp: Slot [12] registered [ 0.263113] acpiphp: Slot [13] registered [ 0.264290] acpiphp: Slot [14] registered [ 0.266113] acpiphp: Slot [15] registered [ 0.267208] acpiphp: Slot [16] registered [ 0.269221] acpiphp: Slot [17] registered [ 0.270105] acpiphp: Slot [18] registered [ 0.272117] acpiphp: Slot [19] registered [ 0.274231] acpiphp: Slot [20] registered [ 0.275106] acpiphp: Slot [21] registered [ 0.277113] acpiphp: Slot [22] registered [ 0.279086] acpiphp: Slot [23] registered [ 0.280096] acpiphp: Slot [24] registered [ 0.282123] acpiphp: Slot [25] registered [ 0.283099] acpiphp: Slot [26] registered [ 0.284097] acpiphp: Slot [27] registered [ 0.285096] acpiphp: Slot [28] registered [ 0.287092] acpiphp: Slot [29] registered [ 0.288060] acpiphp: Slot [30] registered [ 0.289070] acpiphp: Slot [31] registered [ 0.290055] PCI host bridge to bus 0000:00 [ 0.290949] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.295016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300135] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301774] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.304992] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323581] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325847] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.329060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.338018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.351013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355540] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.416878] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.457019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469666] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.492016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.558000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.569017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.580017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.604018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.612324] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.627015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.660025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670148] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.678020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.683016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.706000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708271] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.711262] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.712140] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.715191] iommu: Default domain type: Passthrough [ 0.717340] SCSI subsystem initialized [ 0.718068] ACPI: bus type USB registered [ 0.719068] usbcore: registered new interface driver usbfs [ 0.720038] usbcore: registered new interface driver hub [ 0.721048] usbcore: registered new device driver usb [ 0.722154] pps_core: LinuxPPS API ver. 1 registered [ 0.724012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727073] PTP clock support registered [ 0.729139] EDAC MC: Ver: 3.0.0 [ 0.731148] PCI: Using ACPI for IRQ routing [ 0.732491] NetLabel: Initializing [ 0.734013] NetLabel: domain hash size = 128 [ 0.735009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737084] NetLabel: unlabeled traffic allowed by default [ 0.739140] vgaarb: loaded [ 0.741193] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748348] clocksource: Switched to clocksource kvm-clock [ 0.857974] VFS: Disk quotas dquot_6.6.0 [ 0.859633] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.862427] *** VALIDATE ramfs *** [ 0.863742] *** VALIDATE hugetlbfs *** [ 0.865017] pnp: PnP ACPI init [ 0.867414] pnp: PnP ACPI: found 6 devices [ 0.883143] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.886363] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888751] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.891205] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893579] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895993] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.898676] NET: Registered protocol family 2 [ 0.901107] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.905969] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.909435] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.914494] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.917589] TCP: Hash tables configured (established 65536 bind 65536) [ 0.920575] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.923613] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926668] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929782] NET: Registered protocol family 1 [ 0.932439] RPC: Registered named UNIX socket transport module. [ 0.934677] RPC: Registered udp transport module. [ 0.936529] RPC: Registered tcp transport module. [ 0.938898] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.941578] NET: Registered protocol family 44 [ 0.942893] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.944675] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.946087] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.947976] PCI: CLS 0 bytes, default 64 [ 0.950387] Unpacking initramfs... [ 2.262497] debug: unmapping init [mem 0xffff9cfbfcc54000-0xffff9cfbfffbffff] [ 2.266506] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.269115] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.272447] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.736921] Initialise system trusted keyrings [ 2.738241] Key type blacklist registered [ 2.739489] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.749897] zbud: loaded [ 2.753098] *** VALIDATE nfs *** [ 2.753899] *** VALIDATE nfs4 *** [ 2.755038] pstore: using deflate compression [ 2.757242] Platform Keyring initialized [ 2.841181] NET: Registered protocol family 38 [ 2.842367] Key type asymmetric registered [ 2.843389] Asymmetric key parser 'x509' registered [ 2.844370] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.846250] io scheduler mq-deadline registered [ 2.850610] io scheduler kyber registered [ 2.852062] io scheduler bfq registered [ 2.853644] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.856490] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.859193] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.861876] ACPI: Power Button [PWRF] [ 2.866598] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.872518] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.888168] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.894907] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.910081] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.937770] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.965709] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.969618] Non-volatile memory driver v1.3 [ 2.970764] Linux agpgart interface v0.103 [ 3.001804] virtio_blk virtio1: [vda] 135184 512-byte logical blocks (69.2 MB/66.0 MiB) [ 3.005206] vda: detected capacity change from 0 to 69214208 [ 3.021908] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.024734] vdb: detected capacity change from 0 to 1073741824 [ 3.038648] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.041797] vdc: detected capacity change from 0 to 2621440000 [ 3.056301] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.059238] vdd: detected capacity change from 0 to 2621440000 [ 3.072383] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.074371] vde: detected capacity change from 0 to 4294967296 [ 3.087171] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.089838] vdf: detected capacity change from 0 to 4294967296 [ 3.096764] libphy: Fixed MDIO Bus: probed [ 3.102174] usbcore: registered new interface driver usbserial_generic [ 3.103945] usbserial: USB Serial support registered for generic [ 3.105747] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.109726] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.111220] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.113018] mousedev: PS/2 mouse device common for all mice [ 3.115423] rtc_cmos 00:05: RTC can wake from S4 [ 3.117084] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.118502] rtc_cmos 00:05: registered as rtc0 [ 3.121929] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.125434] intel_pstate: CPU model not supported [ 3.127765] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.130915] hid: raw HID events driver (C) Jiri Kosina [ 3.133353] usbcore: registered new interface driver usbhid [ 3.133880] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.134859] usbhid: USB HID core driver [ 3.134987] drop_monitor: Initializing network drop monitor service [ 3.139768] Initializing XFRM netlink socket [ 3.141461] NET: Registered protocol family 10 [ 3.144313] Segment Routing with IPv6 [ 3.145528] NET: Registered protocol family 17 [ 3.147225] mpls_gso: MPLS GSO support [ 3.151975] RAS: Correctable Errors collector initialized. [ 3.153676] AVX version of gcm_enc/dec engaged. [ 3.154820] AES CTR mode by8 optimization enabled [ 3.219579] sched_clock: Marking stable (3219554636, 0)->(4140391509, -920836873) [ 3.223459] registered taskstats version 1 [ 3.225610] Loading compiled-in X.509 certificates [ 3.227879] zswap: loaded using pool lzo/zbud [ 3.250173] Key type big_key registered [ 3.262148] Key type encrypted registered [ 3.264165] ima: No TPM chip found, activating TPM-bypass! [ 3.265826] ima: Allocated hash algorithm: sha1 [ 3.267132] ima: No architecture policies found [ 3.268510] evm: Initialising EVM extended attributes: [ 3.269819] evm: security.selinux [ 3.270832] evm: security.ima [ 3.271762] evm: security.capability [ 3.272690] evm: HMAC attrs: 0x1 [ 3.275225] rtc_cmos 00:05: setting system clock to 2026-03-30 16:01:56 UTC (1774886516) [ 3.281835] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 3.284172] debug: unmapping init [mem 0xffffffffb1b82000-0xffffffffb1e58fff] [ 3.288103] Write protecting the kernel read-only data: 28672k [ 3.291653] debug: unmapping init [mem 0xffffffffb0203000-0xffffffffb03fffff] [ 3.293874] debug: unmapping init [mem 0xffffffffb0b14000-0xffffffffb0bfffff] [ 3.325672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.332104] systemd[1]: Detected virtualization kvm. [ 3.333435] systemd[1]: Detected architecture x86-64. [ 3.334792] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.358088] systemd[1]: No hostname configured. [ 3.360056] systemd[1]: Set hostname to . [ 3.361566] random: systemd: uninitialized urandom read (16 bytes read) [ 3.363431] systemd[1]: Initializing machine ID from random generator. [ 3.403055] random: ln: uninitialized urandom read (6 bytes read) [ 3.480422] random: systemd: uninitialized urandom read (16 bytes read) [ 3.482986] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.488159] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.494039] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.027877] device-mapper: uevent: version 1.0.3 [ 4.029654] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.682783] virtio_net virtio0 ens2: renamed from eth0 [ 4.709677] random: fast init done [ 4.802336] scsi host0: ata_piix [ 4.844542] scsi host1: ata_piix [ 4.845843] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.848484] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.400780] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.778970] random: crng init done [ 9.780634] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.030313] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.124162] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.334486] SELinux: Disabled at runtime. [ 11.393771] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.399645] systemd[1]: Detected virtualization kvm. [ 11.401066] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.757295] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.759756] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.764157] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.767052] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.770263] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.775837] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.780633] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK [[ 11.825708] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.130310] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.457842] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.490130] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.611832] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.632619] EDAC sbridge: Ver: 1.1.2 [ 14.055260] Key type dns_resolver registered [ 14.349616] NFS: Registering the id_resolver key type [ 14.351500] Key type id_resolver registered [ 14.353213] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg252-server login: [ 39.075986] libcfs: loading out-of-tree module taints kernel. [ 39.095402] Key type ._llcrypt registered [ 39.096773] Key type .llcrypt registered [ 39.143383] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 47.086393] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 47.639684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 47.647495] alg: No test for adler32 (adler32-zlib) [ 48.669200] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 49.008812] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 50.647436] Key type lgssc registered [ 52.601865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.424079] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.462126] hrtimer: interrupt took 8491128 ns [ 122.595971] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 137.862549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.896266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.291291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.350718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.475364] Lustre: lustre-MDT0000: new disk, initializing [ 139.608104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.630124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.322810] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.471499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.582872] Lustre: 6478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.621737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.626180] Lustre: Skipped 1 previous similar message [ 160.771776] Lustre: lustre-MDT0001: new disk, initializing [ 160.929275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.023207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.030810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.044220] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.329755] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 182.920141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.196047] Lustre: lustre-OST0000: new disk, initializing [ 183.206212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.303374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.222292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.237739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.387724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 189.870376] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.471490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.661679] Lustre: lustre-OST0001: new disk, initializing [ 207.664516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.725260] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.633536] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.688095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.717039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.817298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.704857] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.826495] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.739749] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.279861] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing yml_node [ 257.023084] Lustre: DEBUG MARKER: Client: 2.17.50.165 [ 259.942428] Lustre: DEBUG MARKER: MDS: 2.17.50.165 [ 263.056699] Lustre: DEBUG MARKER: OSS: 2.17.50.165 [ 265.041083] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 30 12:06:15 EDT 2026 [ 285.891603] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 287.804681] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 309.732968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.735666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.756837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.240439] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.274059] Lustre: Skipped 2 previous similar messages [ 310.280587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.285505] Lustre: Skipped 3 previous similar messages [ 313.129512] Lustre: server umount lustre-MDT0000 complete [ 318.191348] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774886831 with bad export cookie 5168416108458716879 [ 318.198212] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 318.204303] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.481159] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.483102] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.486736] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.494081] Lustre: Skipped 1 previous similar message [ 320.539740] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 323.488429] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.490034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.530620] Lustre: Skipped 2 previous similar messages [ 325.022861] Lustre: server umount lustre-MDT0001 complete [ 329.362932] Lustre: server umount lustre-OST0000 complete [ 333.835089] Lustre: server umount lustre-OST0001 complete [ 341.419302] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 353.198883] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 394.875773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.239531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.276671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.376407] Lustre: lustre-MDT0000: new disk, initializing [ 395.432933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.444709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.171883] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 417.002833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.241252] Lustre: 14957:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.333778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.344091] Lustre: Skipped 1 previous similar message [ 417.482436] Lustre: lustre-MDT0001: new disk, initializing [ 417.676885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.762639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.780231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 423.027938] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 431.801070] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 436.950375] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 444.289929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.868115] Lustre: lustre-OST0000: new disk, initializing [ 444.879561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 445.037123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.836852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.843257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 447.155357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.967074] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 463.077952] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 468.656220] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.034676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.783746] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 476.133474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 482.784602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 482.792856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.796649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.811382] Lustre: Skipped 1 previous similar message [ 482.820318] Lustre: Skipped 1 previous similar message [ 484.608572] Lustre: server umount lustre-OST0000 complete [ 489.441098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.446062] LustreError: Skipped 1 previous similar message [ 489.450223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.028843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.044665] Lustre: Skipped 2 previous similar messages [ 495.709279] Lustre: server umount lustre-MDT0000 complete [ 498.152910] LustreError: 16194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.172389] LustreError: 16194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 500.015808] LustreError: 14949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887013 with bad export cookie 5168416108458718412 [ 500.042306] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.044063] LustreError: 14949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 500.462725] Lustre: server umount lustre-MDT0001 complete [ 506.249957] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:10:16 (1774887016) [ 509.257316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 511.193707] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:10:22 (1774887022) [ 519.176832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.695713] LustreError: 18964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.810875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.580674] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 525.283418] LustreError: 18964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.015639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.084166] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 545.495709] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 549.578405] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 556.439282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 563.090714] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 568.853601] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 579.556362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.573186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.621926] Lustre: Skipped 1 previous similar message [ 579.659876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.689914] Lustre: Skipped 1 previous similar message [ 583.195784] Lustre: server umount lustre-OST0000 complete [ 593.294581] Lustre: server umount lustre-MDT0000 complete [ 594.914419] LustreError: 20470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.930419] LustreError: 20470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 597.256116] LustreError: 21407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887110 with bad export cookie 5168416108458719294 [ 597.262137] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.272305] LustreError: 21407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 597.752980] Lustre: server umount lustre-MDT0001 complete [ 610.318933] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 613.949588] Key type lgssc unregistered [ 614.281580] LNet: 22145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.287994] LNetError: 22145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.306345] LNet: Removed LNI 192.168.202.152@tcp [ 615.364506] Key type .llcrypt unregistered [ 615.367214] Key type ._llcrypt unregistered [ 640.659442] Key type ._llcrypt registered [ 640.662429] Key type .llcrypt registered [ 640.803604] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 642.084309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.113802] alg: No test for adler32 (adler32-zlib) [ 643.256668] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 643.589036] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 645.319314] Key type lgssc registered [ 647.054430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 657.796890] loop: module loaded [ 657.879405] loop0: detected capacity change from 0 to 204800000 [ 658.145372] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.169158] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.728210] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.454670] loop0: detected capacity change from 0 to 204800000 [ 665.631063] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.641994] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.650940] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 671.706938] loop0: detected capacity change from 0 to 204800000 [ 679.222493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 679.269515] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.751987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 680.956916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 681.157311] Lustre: lustre-MDT0000: new disk, initializing [ 681.271277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.328680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 687.011883] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 696.915189] loop1: detected capacity change from 0 to 204800000 [ 703.514769] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 704.081639] Lustre: lustre-OST07c6: new disk, initializing [ 704.095755] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 704.117441] Lustre: Skipped 1 previous similar message [ 704.353961] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 710.323413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 710.328511] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 710.367180] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 712.252639] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 718.508972] Lustre: 24307:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 718.519365] Lustre: 24307:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 718.532803] Lustre: 24307:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 718.543804] Lustre: 24307:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 718.559929] Lustre: 24307:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 718.578234] Lustre: 24307:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 742.176596] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 742.203358] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.215229] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.222431] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 742.225494] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 742.237091] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.176388] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 747.185732] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.190266] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 747.195341] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.203467] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.211632] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.225535] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 747.237183] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.244631] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 747.249951] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.259048] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.271872] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.182610] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 752.196640] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.204736] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 752.213199] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.223359] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.236657] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.243783] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 752.251366] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.256394] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.261381] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.266316] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.269715] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.993602] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 760.009510] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 760.016616] Lustre: 24308:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 760.030087] Lustre: 24308:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.033677] Lustre: 24308:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.037331] Lustre: 24308:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.040652] Lustre: 24308:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 760.044667] Lustre: 24308:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.048525] Lustre: 24308:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 760.052443] Lustre: 24308:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.055576] Lustre: 24308:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.059300] Lustre: 24308:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 771.431355] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 771.452456] Lustre: 25954:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 771.467988] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 771.502468] Lustre: 25954:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.511580] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.524873] Lustre: 25954:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.528506] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 771.545883] Lustre: 25954:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.561114] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 771.570932] Lustre: 25954:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.580225] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.588229] Lustre: 25954:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.635144] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.650445] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 822.752867] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.342826] Lustre: server umount lustre-OST07c6 complete [ 833.865543] Lustre: server umount lustre-MDT0000 complete [ 852.464470] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 855.473807] Key type lgssc unregistered [ 855.742705] LNet: 27827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.755541] LNetError: 27827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.782696] LNet: Removed LNI 192.168.202.152@tcp [ 856.777041] Key type .llcrypt unregistered [ 856.780433] Key type ._llcrypt unregistered [ 877.262880] Key type ._llcrypt registered [ 877.266289] Key type .llcrypt registered [ 877.384612] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 878.866880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.023411] alg: No test for adler32 (adler32-zlib) [ 880.358336] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 880.717293] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 882.447345] Key type lgssc registered [ 883.925880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.185989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.236700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.799034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.817807] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 903.143517] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.158422] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 908.264121] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.661375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.034173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 914.711879] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 921.534794] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 926.252478] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 934.108434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.656928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.751602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.469550] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 949.373912] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 974.819933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 974.842083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.870955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.941587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.960097] Lustre: Skipped 2 previous similar messages [ 985.058886] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 985.065724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 985.090946] Lustre: Skipped 1 previous similar message [ 985.296498] Lustre: server umount lustre-OST0000 complete [ 991.203739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.226882] LustreError: Skipped 1 previous similar message [ 991.229127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.246160] Lustre: Skipped 1 previous similar message [ 991.251642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.300823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.308262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.404165] Lustre: server umount lustre-MDT0000 complete [ 1000.418114] LustreError: 30194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.437604] LustreError: 30194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1001.731472] LustreError: 29062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887514 with bad export cookie 2739883134375591148 [ 1001.745189] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.185671] Lustre: server umount lustre-MDT0001 complete [ 1013.829984] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1017.207868] Key type lgssc unregistered [ 1017.555537] LNet: 32474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.562226] LNetError: 32474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.606799] LNet: Removed LNI 192.168.202.152@tcp [ 1019.625396] Key type .llcrypt unregistered [ 1019.626969] Key type ._llcrypt unregistered [ 1040.565968] Key type ._llcrypt registered [ 1040.568148] Key type .llcrypt registered [ 1040.650205] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1041.539201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.733871] alg: No test for adler32 (adler32-zlib) [ 1042.915486] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1043.147644] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1044.783205] Key type lgssc registered [ 1045.918183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.352657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.370188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.106806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.213673] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1067.492737] LustreError: 33727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.512386] LustreError: 33727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1072.471350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.622914] LustreError: 33727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.032567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.192692] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1086.263793] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1090.779170] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1098.059833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1098.686593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.787928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1106.156372] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1112.647889] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1123.308649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1123.340696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.351031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1123.355613] Lustre: Skipped 1 previous similar message [ 1123.367067] Lustre: Skipped 1 previous similar message [ 1128.430260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.479439] Lustre: Skipped 1 previous similar message [ 1128.612847] Lustre: server umount lustre-OST0000 complete [ 1133.549051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.554482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.559111] Lustre: Skipped 1 previous similar message [ 1133.581464] Lustre: Skipped 1 previous similar message [ 1138.579358] Lustre: server umount lustre-MDT0000 complete [ 1138.664491] LustreError: 33723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.679871] LustreError: 33723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1143.313635] LustreError: 35975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887656 with bad export cookie 13464418878456537046 [ 1143.315761] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.325229] LustreError: 35975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.844883] Lustre: server umount lustre-MDT0001 complete [ 1156.754829] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1159.663286] Key type lgssc unregistered [ 1159.968204] LNet: 36911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.978755] LNetError: 36911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.998604] LNet: Removed LNI 192.168.202.152@tcp [ 1160.857315] Key type .llcrypt unregistered [ 1160.859610] Key type ._llcrypt unregistered [ 1187.506340] Key type ._llcrypt registered [ 1187.507335] Key type .llcrypt registered [ 1187.547806] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1188.088347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1188.102162] alg: No test for adler32 (adler32-zlib) [ 1189.122198] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1189.256921] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1190.994084] Key type lgssc registered [ 1192.831513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.819585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.872344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.440764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.870354] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1213.928621] LustreError: 38313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.960555] LustreError: 38313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1219.040633] LustreError: 38313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.537910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.861357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1226.092317] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1233.522825] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1238.023675] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1246.200100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.726820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.794075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1254.055942] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1260.736886] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1268.686904] Lustre: server umount lustre-MDT0000 complete [ 1268.963388] LustreError: 39375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.246383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.282944] Lustre: Skipped 2 previous similar messages [ 1273.062620] LustreError: 38293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887786 with bad export cookie 7300820758946535378 [ 1273.063667] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.079279] LustreError: 38293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1274.056298] LustreError: 38309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.080271] LustreError: 38309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1274.086635] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 1275.363350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.377767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1278.370683] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.371683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1278.400032] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1279.943946] Lustre: server umount lustre-MDT0001 complete [ 1302.484385] Lustre: server umount lustre-OST0000 complete [ 1325.471210] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1329.158046] Key type lgssc unregistered [ 1329.557278] LNet: 41821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.562039] LNetError: 41821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.582198] LNet: Removed LNI 192.168.202.152@tcp [ 1330.419384] Key type .llcrypt unregistered [ 1330.421472] Key type ._llcrypt unregistered [ 1360.355463] Key type ._llcrypt registered [ 1360.358251] Key type .llcrypt registered [ 1360.553892] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1362.034052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.254324] alg: No test for adler32 (adler32-zlib) [ 1363.356651] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1363.682672] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1365.424880] Key type lgssc registered [ 1366.837461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1376.022498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1376.048476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.796610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.708656] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1386.980392] LustreError: 43226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.999781] LustreError: 43226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1392.102820] LustreError: 43226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.375892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.837167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.128083] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1406.932768] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1411.921690] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1420.555209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.000793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.016186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1428.325864] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1434.473781] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1447.645032] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1447.927410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.930549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.953818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.956385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.979286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.020836] Lustre: Skipped 2 previous similar messages [ 1452.742976] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1455.075833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.086874] Lustre: Skipped 1 previous similar message [ 1459.167203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.589391] Lustre: server umount lustre-OST0000 complete [ 1480.674423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.684243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.689434] Lustre: Skipped 2 previous similar messages [ 1486.170969] Lustre: server umount lustre-MDT0000 complete [ 1486.832417] LustreError: 43225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.863466] LustreError: 43225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1491.247578] LustreError: 43206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888004 with bad export cookie 12160642353779386817 [ 1491.255634] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.635809] Lustre: server umount lustre-MDT0001 complete [ 1511.093792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1514.843790] Key type lgssc unregistered [ 1515.139834] LNet: 46585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.149338] LNetError: 46585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.164344] LNet: Removed LNI 192.168.202.152@tcp [ 1516.387709] Key type .llcrypt unregistered [ 1516.391628] Key type ._llcrypt unregistered [ 1546.954669] Key type ._llcrypt registered [ 1546.961968] Key type .llcrypt registered [ 1547.153745] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1548.804879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.863246] alg: No test for adler32 (adler32-zlib) [ 1550.277337] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1550.517488] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1552.311256] Key type lgssc registered [ 1553.565830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1563.003676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1563.055191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.863936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.132355] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1575.395338] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.419980] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1579.514053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.076187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1585.676581] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1592.315675] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.853318] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1605.896312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.465594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.550959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1614.054832] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1620.698324] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1629.420542] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1633.149559] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1645.826601] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-3012h29m15s [ 1647.496990] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1650.661194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.667073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.682967] Lustre: Skipped 2 previous similar messages [ 1650.704424] Lustre: Skipped 2 previous similar messages [ 1651.746909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.765808] Lustre: Skipped 2 previous similar messages [ 1652.975160] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1656.045819] Lustre: server umount lustre-MDT0000 complete [ 1657.833064] LustreError: 48696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.865178] LustreError: 48696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1660.106765] LustreError: 47982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.114236] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1663.986727] LustreError: 47986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1664.024689] LustreError: 47986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1668.261308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.435264] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.713572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.815367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1669.739464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1673.922805] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1678.199943] Lustre: DEBUG MARKER: Wait for df (36176) ... [ 1680.114644] Lustre: DEBUG MARKER: done [ 1685.478928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.497591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.498456] Lustre: Skipped 1 previous similar message [ 1685.517331] Lustre: Skipped 2 previous similar messages [ 1690.594184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.605536] Lustre: Skipped 1 previous similar message [ 1699.295130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1699.478237] Lustre: server umount lustre-OST0000 complete [ 1705.973542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.980146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1705.988691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.988708] Lustre: Skipped 2 previous similar messages [ 1706.001997] Lustre: Skipped 1 previous similar message [ 1710.695251] Lustre: server umount lustre-MDT0000 complete [ 1713.123064] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.150723] LustreError: 47985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1715.673977] LustreError: 47965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888228 with bad export cookie 2265959171198339255 [ 1715.689211] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.695706] LustreError: 47965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1716.173816] Lustre: server umount lustre-MDT0001 complete [ 1728.445756] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1731.644427] Key type lgssc unregistered [ 1731.987407] LNet: 52548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.993579] LNetError: 52548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.017846] LNet: Removed LNI 192.168.202.152@tcp [ 1732.704756] Key type .llcrypt unregistered [ 1732.710953] Key type ._llcrypt unregistered [ 1780.075708] Key type ._llcrypt registered [ 1780.078751] Key type .llcrypt registered [ 1780.232803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1782.525516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.563904] alg: No test for adler32 (adler32-zlib) [ 1783.672412] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1784.053836] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1785.815280] Key type lgssc registered [ 1787.961835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1798.632625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1798.646862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.198608] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1800.311827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1800.471663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.610847] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1815.533508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.643825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1815.698316] Lustre: 54746:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1815.997642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.405063] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1826.550510] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1830.364557] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1837.445850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.666394] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1837.932801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.763503] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1845.960323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1850.488343] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1858.383871] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-3012h32m49s [ 1860.080846] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1867.470207] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1869.999973] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1870.066152] LustreError: 54752:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1890.541588] Lustre: lustre-MDT0000: Client 710c8530-b3b0-4736-8a88-f3e84e49326c (at 192.168.202.52@tcp) reconnecting [ 1910.079120] LustreError: 54752:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1913.066146] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1915.708956] Lustre: DEBUG MARKER: done [ 1923.046743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.048902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1923.049888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.074866] Lustre: Skipped 1 previous similar message [ 1923.095800] LustreError: Skipped 1 previous similar message [ 1928.169805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.182373] Lustre: Skipped 1 previous similar message [ 1933.299687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.311772] Lustre: Skipped 2 previous similar messages [ 1936.351615] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1936.648032] Lustre: server umount lustre-OST0000 complete [ 1943.520970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.528497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.545951] Lustre: Skipped 1 previous similar message [ 1943.568669] Lustre: Skipped 1 previous similar message [ 1946.787045] Lustre: server umount lustre-MDT0000 complete [ 1950.540331] LustreError: 56379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888463 with bad export cookie 16485216730847011653 [ 1950.545313] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.557374] LustreError: 56379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1950.864861] Lustre: server umount lustre-MDT0001 complete [ 1961.865606] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1964.938817] Key type lgssc unregistered [ 1965.376832] LNet: 58723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.386495] LNetError: 58723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.438632] LNet: Removed LNI 192.168.202.152@tcp [ 1967.391556] Key type .llcrypt unregistered [ 1967.393567] Key type ._llcrypt unregistered [ 2013.156811] Key type ._llcrypt registered [ 2013.158243] Key type .llcrypt registered [ 2013.326759] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2014.375674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2014.451712] alg: No test for adler32 (adler32-zlib) [ 2015.680330] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2015.902571] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2017.592351] Key type lgssc registered [ 2019.322698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.962232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2023.977067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.442332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2025.487418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2025.758436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.879878] Lustre: Failing over lustre-MDT0000 [ 2027.486236] Lustre: server umount lustre-MDT0000 complete [ 2034.946750] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:35:45 (1774888545) [ 2041.748819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.199267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.674104] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2055.848662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.962741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2055.991347] Lustre: 61506:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2056.119110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2056.317528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2061.838286] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2068.090035] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2072.956124] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.429144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.625990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2079.883826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.110464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2086.753943] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2093.077029] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2099.619453] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2106.851896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.861246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2106.867905] Lustre: Skipped 1 previous similar message [ 2106.885836] Lustre: Skipped 2 previous similar messages [ 2111.969639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.977504] Lustre: Skipped 2 previous similar messages [ 2112.336769] Lustre: server umount lustre-MDT0000 complete [ 2116.507365] LustreError: 62225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888629 with bad export cookie 7600410438928866469 [ 2116.518974] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.520386] LustreError: 62225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2117.031730] Lustre: server umount lustre-MDT0001 complete [ 2120.941368] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2125.835949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2133.472187] Lustre: 60430:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888630/real 1774888630] req@000000000fea4bef x1861105514224640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2133.526487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.548957] Lustre: Skipped 1 previous similar message [ 2135.095972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.695448] Lustre: 60430:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888634/real 1774888634] req@0000000015560615 x1861105514224768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.215703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.376147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2144.224351] Lustre: 60431:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888639/real 1774888639] req@00000000af1ece49 x1861105514225152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.261588] Lustre: 60431:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2146.621941] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2147.298524] Lustre: 60429:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888644/real 1774888644] req@00000000b34a651f x1861105514225280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2148.333790] LustreError: 63128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.356527] LustreError: 63128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2153.442660] LustreError: 63128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.481498] LustreError: 63128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2155.876928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.586283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2161.557370] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2161.658185] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2162.693292] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2167.639529] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2172.658896] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2182.126651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.133581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.143238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.208407] Lustre: server umount lustre-MDT0000 complete [ 2188.268791] LustreError: 64936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.296660] LustreError: 64936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2191.614173] LustreError: 62225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888704 with bad export cookie 7600410438928868744 [ 2191.632929] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2191.637227] LustreError: 62225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2192.010094] Lustre: server umount lustre-MDT0001 complete [ 2199.750528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2209.759290] Lustre: 60430:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888706/real 1774888706] req@0000000045a414a8 x1861105514252288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2209.793866] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.814798] Lustre: Skipped 2 previous similar messages [ 2210.271751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.620840] LustreError: 63128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.773171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.908462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2221.858475] Lustre: 60430:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888719/real 1774888719] req@00000000234fc48d x1861105514252928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2221.894650] Lustre: 60430:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2222.424752] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2225.130711] LustreError: 67394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.158808] LustreError: 67394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2232.061951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.800514] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2238.479809] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2244.932301] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2249.317509] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2258.405976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2258.419424] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.429312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.429320] Lustre: Skipped 3 previous similar messages [ 2258.481764] Lustre: Skipped 1 previous similar message [ 2264.239076] Lustre: server umount lustre-MDT0000 complete [ 2268.645207] LustreError: 67767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.678178] LustreError: 67767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2270.937068] LustreError: 61498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888784 with bad export cookie 7600410438928870123 [ 2270.952448] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2270.982556] LustreError: 61498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2272.019102] Lustre: server umount lustre-MDT0001 complete [ 2282.757746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2287.689911] Lustre: DEBUG MARKER: files compared the same [ 2289.119107] Lustre: 60432:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774888786/real 1774888786] req@0000000045efafff x1861105514278272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774888802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2289.153387] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.177016] Lustre: Skipped 1 previous similar message [ 2289.823771] Lustre: server umount lustre-OST0000 complete [ 2308.980833] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2312.395125] Key type lgssc unregistered [ 2312.748640] LNet: 70430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.758734] LNetError: 70430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.784753] LNet: Removed LNI 192.168.202.152@tcp [ 2313.635561] Key type .llcrypt unregistered [ 2313.638658] Key type ._llcrypt unregistered [ 2342.513189] Key type ._llcrypt registered [ 2342.515430] Key type .llcrypt registered [ 2342.620934] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2343.701234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.994067] alg: No test for adler32 (adler32-zlib) [ 2345.025213] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2345.188577] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2346.847183] Key type lgssc registered [ 2348.751774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2358.000621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2358.028351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.737229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.556835] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 2368.996433] LustreError: 71831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.012558] LustreError: 71831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2373.842891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.112305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2374.324043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.262555] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 2386.109274] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2391.209689] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2399.124530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.208673] LustreError: 73426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.378677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.555784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2405.501873] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 2412.044804] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2425.312474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2425.324731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.325625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.337457] LustreError: Skipped 1 previous similar message [ 2425.360952] Lustre: Skipped 1 previous similar message [ 2430.433232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.438842] Lustre: Skipped 1 previous similar message [ 2435.042480] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2435.187322] Lustre: server umount lustre-OST0000 complete [ 2440.671812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.683107] Lustre: Skipped 1 previous similar message [ 2440.687020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.501565] Lustre: server umount lustre-MDT0000 complete [ 2445.792165] LustreError: 71831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.809952] LustreError: 71831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2450.109135] LustreError: 71813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888963 with bad export cookie 4845094839546740461 [ 2450.118809] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.130211] LustreError: 71813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2450.668996] Lustre: server umount lustre-MDT0001 complete [ 2463.751972] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2466.818405] Key type lgssc unregistered [ 2467.175438] LNet: 75012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.183567] LNetError: 75012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.207883] LNet: Removed LNI 192.168.202.152@tcp [ 2468.274417] Key type .llcrypt unregistered [ 2468.277894] Key type ._llcrypt unregistered [ 2495.452211] Key type ._llcrypt registered [ 2495.456925] Key type .llcrypt registered [ 2495.573949] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2496.762146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2496.909092] alg: No test for adler32 (adler32-zlib) [ 2498.147375] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2498.379637] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2500.087178] Key type lgssc registered [ 2501.367656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2510.109203] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2510.128791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2525.792527] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2530.911427] LustreError: 76386:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 2557.664329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.509922] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2570.638231] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2578.227594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.307909] Lustre: *** cfs_fail_loc=706, val=0*** [ 2578.606279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.658731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2583.100760] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2584.034358] LustreError: 76411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.057971] LustreError: 76411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2589.158833] LustreError: 76411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.231385] LustreError: 76411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.258967] LustreError: 76411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2592.807164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.421466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.673025] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2604.665300] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2608.957472] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2612.206691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.231514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2612.235682] Lustre: Skipped 1 previous similar message [ 2617.322332] LustreError: 76402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.346956] LustreError: 76402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2617.472240] Lustre: server umount lustre-OST0000 complete [ 2622.435518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.437743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2622.459241] Lustre: Skipped 2 previous similar messages [ 2622.481724] Lustre: Skipped 1 previous similar message [ 2627.542227] Lustre: server umount lustre-MDT0000 complete [ 2627.567335] LustreError: 77357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.582745] LustreError: 77357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2631.152579] LustreError: 76393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889144 with bad export cookie 1963878959891235794 [ 2631.162264] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2631.536943] Lustre: server umount lustre-MDT0001 complete [ 2642.921793] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2645.746968] Key type lgssc unregistered [ 2646.087615] LNet: 79649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.092357] LNetError: 79649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.105510] LNet: Removed LNI 192.168.202.152@tcp [ 2647.149554] Key type .llcrypt unregistered [ 2647.151533] Key type ._llcrypt unregistered [ 2671.590771] Key type ._llcrypt registered [ 2671.598073] Key type .llcrypt registered [ 2671.783970] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2673.356689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2673.430429] alg: No test for adler32 (adler32-zlib) [ 2674.504225] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2674.721234] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2676.415190] Key type lgssc registered [ 2677.753676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2684.897780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2684.923760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.465487] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2697.925784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.286485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2702.775377] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2703.330143] LustreError: 81718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.354499] LustreError: 81718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2708.451668] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.463424] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2711.267731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.561485] LustreError: 81719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.574381] LustreError: 81719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2711.595414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.642462] LustreError: 81719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.656080] LustreError: 81719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2717.941860] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2721.768841] LustreError: 81718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.789399] LustreError: 81718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2726.267823] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.281379] Lustre: MGS: Client d2afb290-0137-4296-b934-6b29ff7d0b45 (at 0@lo) reconnecting [ 2726.296624] LustreError: 81733:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2726.508805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.574514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2731.222367] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2746.853213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2746.866108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.883632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2751.714644] Lustre: server umount lustre-OST0000 complete [ 2757.603444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.610935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2757.617920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.617932] Lustre: Skipped 2 previous similar messages [ 2757.628474] Lustre: Skipped 2 previous similar messages [ 2761.816235] Lustre: server umount lustre-MDT0000 complete [ 2762.724684] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.758195] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2766.895361] Lustre: server umount lustre-MDT0001 complete [ 2771.760767] Lustre: server umount MGS complete [ 2783.532814] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:48:14 (1774889294) [ 2809.874633] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2817.795804] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2848.770685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.781718] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2863.397681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.627307] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2863.675946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2863.766610] Lustre: lustre-MDT0001: new disk, initializing [ 2863.865578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.888657] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.900382] Lustre: 88810:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2863.916814] LustreError: 88810:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2867.629598] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2876.808746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.059396] Lustre: lustre-OST0000: new disk, initializing [ 2877.063040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2877.125821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.365721] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2885.153576] Lustre: 89998:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000dcc5bcbf x1861106204188928/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1774889414 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2889.338373] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2895.780216] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2895.791978] Lustre: MGS: Client 22b579e1-0f45-4c85-8573-d0a3667b6dfd (at 0@lo) reconnecting [ 2895.843528] Lustre: 88096:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2895.867963] LustreError: 88096:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2895.879967] LustreError: 88096:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2895.906594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2895.975674] Lustre: lustre-MDT0000: new disk, initializing [ 2896.119296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.144834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2896.260690] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2897.185122] Lustre: 90517:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000c7778c51 x1861106204195456/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1774889426 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2899.961777] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2905.578761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2905.598891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2906.298287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2906.418157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2928.611555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.614928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.639591] Lustre: Skipped 1 previous similar message [ 2928.652110] Lustre: Skipped 2 previous similar messages [ 2933.732317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.393502] Lustre: server umount lustre-OST0000 complete [ 2938.854968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.866457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2938.877630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.894435] Lustre: Skipped 2 previous similar messages [ 2943.693900] Lustre: server umount lustre-MDT0000 complete [ 2943.969920] LustreError: 90860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.985759] LustreError: 90860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2953.610750] Lustre: server umount lustre-MDT0001 complete [ 2956.837333] Lustre: server umount MGS complete [ 2965.083329] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:51:16 (1774889476) [ 2979.400079] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2982.017932] Key type lgssc unregistered [ 2982.277817] LNet: 92982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2982.284253] LNetError: 92982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2982.291737] LNet: Removed LNI 192.168.202.152@tcp [ 2983.139473] Key type .llcrypt unregistered [ 2983.141813] Key type ._llcrypt unregistered [ 2998.563447] Key type ._llcrypt registered [ 2998.565257] Key type .llcrypt registered [ 2998.641072] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2999.496608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2999.577589] alg: No test for adler32 (adler32-zlib) [ 3000.562568] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3000.779743] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3002.407377] Key type lgssc registered [ 3003.231812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3009.325067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3009.339503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.858618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.152735] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3020.679786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.980973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.855975] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3029.499768] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3032.322648] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3037.990109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.215564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.356480] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3048.568836] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3057.720753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3066.852792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3066.858792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.876442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.874413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.878084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.879811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3073.004438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3073.013723] Lustre: Skipped 2 previous similar messages [ 3078.115950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.126507] Lustre: Skipped 1 previous similar message [ 3079.135571] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3079.298243] Lustre: server umount lustre-OST0000 complete [ 3082.724936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3082.733452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.755591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.832432] Lustre: server umount lustre-MDT0000 complete [ 3091.946934] LustreError: 94170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889605 with bad export cookie 3963905547540983993 [ 3091.959138] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.241975] Lustre: server umount lustre-MDT0001 complete [ 3101.154899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3103.368317] Key type lgssc unregistered [ 3103.618031] LNet: 97389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3103.621806] LNetError: 97389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3103.634313] LNet: Removed LNI 192.168.202.152@tcp [ 3104.249878] Key type .llcrypt unregistered [ 3104.251603] Key type ._llcrypt unregistered [ 3116.414566] Key type ._llcrypt registered [ 3116.416364] Key type .llcrypt registered [ 3116.501134] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3117.406044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3117.428350] alg: No test for adler32 (adler32-zlib) [ 3118.478429] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3118.703648] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3120.327955] Key type lgssc registered [ 3121.184595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.367678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.378184] LustreError: 98430:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3125.382996] LustreError: 98436:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3125.388125] LustreError: 98430:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3125.390607] LustreError: 98430:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3125.399885] LustreError: 98430:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3125.414113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.872276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.544195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.796124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3136.054843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3139.957804] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing lsmod [ 3142.167277] LustreError: 99606:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3142.168112] LustreError: 99605:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3142.177093] LustreError: 99606:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3142.182300] LustreError: 99606:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3142.191882] LustreError: 99606:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3142.196690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.531762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.755754] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.761837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.777194] Lustre: Skipped 2 previous similar messages [ 3147.787685] Lustre: Skipped 2 previous similar messages [ 3151.091798] Lustre: server umount lustre-MDT0000 complete [ 3152.865588] LustreError: 98473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.892712] LustreError: 98473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3153.966311] LustreError: 98455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889667 with bad export cookie 17870630306525285351 [ 3153.968590] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3153.972643] LustreError: 98455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3154.177432] Lustre: server umount lustre-MDT0001 complete [ 3167.360897] Lustre: server umount lustre-OST0000 complete [ 3173.085667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.445619] LustreError: 100673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.517750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.362915] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3178.977467] LustreError: 100673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.419473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.695720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3185.780641] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3189.459147] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3192.180059] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3197.592327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.844347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.899435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3201.830200] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3205.539782] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3217.379829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.381958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.384849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.393716] Lustre: Skipped 1 previous similar message [ 3217.417150] Lustre: Skipped 1 previous similar message [ 3221.124884] Lustre: server umount lustre-OST0000 complete [ 3227.617052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.618960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.627558] Lustre: Skipped 1 previous similar message [ 3227.635733] Lustre: Skipped 1 previous similar message [ 3232.736612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.743358] Lustre: Skipped 1 previous similar message [ 3236.833195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.840871] Lustre: Skipped 1 previous similar message [ 3238.367161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.556943] Lustre: server umount lustre-MDT0000 complete [ 3241.286278] LustreError: 100652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889754 with bad export cookie 17870630306525286030 [ 3241.290362] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.301033] LustreError: 100652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3241.602995] Lustre: server umount lustre-MDT0001 complete [ 3249.464818] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3251.459553] Key type lgssc unregistered [ 3251.693303] LNet: 103847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.698916] LNetError: 103847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.715787] LNet: Removed LNI 192.168.202.152@tcp [ 3252.362737] Key type .llcrypt unregistered [ 3252.365435] Key type ._llcrypt unregistered [ 3269.413246] Key type ._llcrypt registered [ 3269.414822] Key type .llcrypt registered [ 3269.503327] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3270.223303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3270.240341] alg: No test for adler32 (adler32-zlib) [ 3271.163089] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3271.332931] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3272.976845] Key type lgssc registered [ 3273.706732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3279.347054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3279.377301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.699050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.589829] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3290.081629] LustreError: 105228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.090557] LustreError: 105228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3290.110160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.318942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3293.085545] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3296.900635] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3299.614302] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3304.318197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.623992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.649636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3309.090725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3312.837935] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3317.750824] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3320.160361] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3325.409495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3325.411132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.414304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3325.420565] Lustre: Skipped 1 previous similar message [ 3328.833307] Lustre: server umount lustre-OST0000 complete [ 3331.554161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3331.574441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.587782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.593114] Lustre: Skipped 1 previous similar message [ 3335.656724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3335.662325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.545438] Lustre: server umount lustre-MDT0000 complete [ 3340.303740] LustreError: 105208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889853 with bad export cookie 16227512349021718931 [ 3340.304614] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3340.313684] LustreError: 105208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3340.547408] Lustre: server umount lustre-MDT0001 complete [ 3348.133377] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3350.309762] Key type lgssc unregistered [ 3350.507667] LNet: 108505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3350.516603] LNetError: 108505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3350.535320] LNet: Removed LNI 192.168.202.152@tcp [ 3351.109908] Key type .llcrypt unregistered [ 3351.111987] Key type ._llcrypt unregistered [ 3364.162926] Key type ._llcrypt registered [ 3364.165278] Key type .llcrypt registered [ 3364.236327] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3365.037752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.160414] alg: No test for adler32 (adler32-zlib) [ 3366.134716] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3366.276214] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3367.903195] Key type lgssc registered [ 3368.731237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.911204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3374.927824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.320561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.142125] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3384.985346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.280629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.225200] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3392.329267] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3394.842796] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3399.409132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.660594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.739021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3403.471859] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3407.282725] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3417.058279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.064368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.073819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.128758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3420.130783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.139581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.155329] Lustre: Skipped 1 previous similar message [ 3422.378981] Lustre: server umount lustre-OST0000 complete [ 3425.249393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.251137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.262174] Lustre: Skipped 1 previous similar message [ 3425.273762] Lustre: Skipped 1 previous similar message [ 3430.369524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.375449] Lustre: Skipped 1 previous similar message [ 3431.221770] Lustre: server umount lustre-MDT0000 complete [ 3433.886292] LustreError: 111842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889947 with bad export cookie 17377553011169161462 [ 3433.887300] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.897034] LustreError: 111842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3434.096789] Lustre: server umount lustre-MDT0001 complete [ 3441.336706] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3443.309534] Key type lgssc unregistered [ 3443.501634] LNet: 112867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3443.506507] LNetError: 112867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3443.523299] LNet: Removed LNI 192.168.202.152@tcp [ 3444.116328] Key type .llcrypt unregistered [ 3444.117878] Key type ._llcrypt unregistered [ 3460.252272] Key type ._llcrypt registered [ 3460.253991] Key type .llcrypt registered [ 3460.326516] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3460.993366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3461.074944] alg: No test for adler32 (adler32-zlib) [ 3462.032562] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3462.177641] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3463.783163] Key type lgssc registered [ 3464.591715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3469.975396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3469.995248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.366232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.570329] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3480.415984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.674761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3484.185585] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3488.145262] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.895330] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3495.818272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3495.970271] LustreError: 115840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.979855] LustreError: 115840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3496.055116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.181792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3501.552329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3504.666252] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3509.839617] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3512.568571] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3512.573945] Lustre: Skipped 2 previous similar messages [ 3515.688644] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3515.692773] Lustre: Skipped 2 previous similar messages [ 3518.540153] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3518.579608] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3523.500279] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3523.508255] Lustre: Skipped 2 previous similar messages [ 3525.238640] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp [ 3531.358370] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.52/24@tcp in log lustre-MDT0000 [ 3531.367614] Lustre: Skipped 2 previous similar messages [ 3537.956905] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.52/24@tcp [ 3537.968312] Lustre: Skipped 1 previous similar message [ 3542.496821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3542.506731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.526392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3546.594532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3546.595238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.446381] Lustre: server umount lustre-OST0000 complete [ 3551.713714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.717102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.725650] Lustre: Skipped 1 previous similar message [ 3551.738504] Lustre: Skipped 2 previous similar messages [ 3556.833174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3556.843939] Lustre: Skipped 1 previous similar message [ 3557.570835] Lustre: server umount lustre-MDT0000 complete [ 3560.508666] LustreError: 114225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890073 with bad export cookie 16015231138693860244 [ 3560.509835] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.515300] LustreError: 114225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3560.868395] Lustre: server umount lustre-MDT0001 complete [ 3569.106562] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3571.249551] Key type lgssc unregistered [ 3571.496911] LNet: 118789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3571.503598] LNetError: 118789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3571.515310] LNet: Removed LNI 192.168.202.152@tcp [ 3572.092652] Key type .llcrypt unregistered [ 3572.095199] Key type ._llcrypt unregistered [ 3593.228021] Key type ._llcrypt registered [ 3593.230037] Key type .llcrypt registered [ 3593.313521] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3593.860644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3593.897735] alg: No test for adler32 (adler32-zlib) [ 3594.915616] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3595.117058] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3596.767280] Key type lgssc registered [ 3597.606575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3603.493332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3603.520412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.878897] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3604.903694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.905252] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.52/24@tcp [ 3607.841931] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3614.554832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.762321] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3614.774540] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.52/24@tcp [ 3614.804212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3618.063743] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3621.976579] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3624.661875] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3628.874163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3629.128685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.209533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3632.205848] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3635.278376] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3646.948426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.951744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.972105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.018304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.020133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.029725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.831201] Lustre: server umount lustre-OST0000 complete [ 3655.139862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.141994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.149231] Lustre: Skipped 1 previous similar message [ 3655.154018] Lustre: Skipped 1 previous similar message [ 3660.257176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.264761] Lustre: Skipped 1 previous similar message [ 3660.493343] Lustre: server umount lustre-MDT0000 complete [ 3662.961225] LustreError: 120293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890176 with bad export cookie 2423061998094280856 [ 3662.966276] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3662.971087] LustreError: 120293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3663.250131] Lustre: server umount lustre-MDT0001 complete [ 3669.226319] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3670.899831] Key type lgssc unregistered [ 3671.090291] LNet: 123543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3671.097618] LNetError: 123543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3671.113079] LNet: Removed LNI 192.168.202.152@tcp [ 3671.672520] Key type .llcrypt unregistered [ 3671.680916] Key type ._llcrypt unregistered [ 3687.361674] Key type ._llcrypt registered [ 3687.363654] Key type .llcrypt registered [ 3687.441848] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3695.796634] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3696.378596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3696.455127] alg: No test for adler32 (adler32-zlib) [ 3697.395220] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3697.573248] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3699.191169] Key type lgssc registered [ 3699.908421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3719.930458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3719.952378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.134184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3721.164829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3721.226551] Lustre: lustre-MDT0000: new disk, initializing [ 3721.283127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.300961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3723.281264] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3730.242166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.328478] Lustre: 126884:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3730.359798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3730.361983] Lustre: Skipped 1 previous similar message [ 3730.419776] Lustre: lustre-MDT0001: new disk, initializing [ 3730.493234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3730.513541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3730.528609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3733.532257] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3738.903691] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3741.442179] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3745.304238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.547733] Lustre: lustre-OST0000: new disk, initializing [ 3745.554532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3745.649820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.973258] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3753.837224] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3755.028502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3755.034267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3755.090187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3756.834132] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3756.970745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3759.528799] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3759.658591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3762.998577] Lustre: server umount lustre-OST0000 complete [ 3765.217381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.221224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3765.222915] Lustre: Skipped 1 previous similar message [ 3770.338195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.349791] Lustre: Skipped 2 previous similar messages [ 3775.461179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3775.463839] Lustre: Skipped 1 previous similar message [ 3775.515535] Lustre: server umount lustre-MDT0000 complete [ 3777.457912] LustreError: 128636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890290 with bad export cookie 7393001304014651039 [ 3777.463764] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.468463] LustreError: 128636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3787.816529] Lustre: server umount lustre-MDT0001 complete [ 3792.113358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.378685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.536396] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3797.477208] LustreError: 130694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.484129] LustreError: 130694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3799.898086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.136310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.528561] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3805.339984] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3807.168924] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3811.119133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.316534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.449456] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3817.307663] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3819.231062] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3819.334645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3821.347925] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3826.599805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 5 sec [ 3832.373360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3832.474025] Lustre: lustre-OST0001: new disk, initializing [ 3832.480110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3832.526112] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3835.493373] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3839.745359] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3840.529672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3840.534946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3840.595141] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3840.873718] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3842.889388] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3842.990823] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3844.713662] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ca9c974f000.ost_server_uuid 50 [ 3845.380737] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ca9c974f000.ost_server_uuid in FULL state after 0 sec [ 3850.725178] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.726646] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.728158] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3850.731206] Lustre: Skipped 1 previous similar message [ 3850.742448] LustreError: Skipped 1 previous similar message [ 3854.550181] Lustre: server umount lustre-OST0001 complete [ 3855.846497] LustreError: 132287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.857430] LustreError: 132287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3856.352203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3856.357649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.365173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.366931] Lustre: Skipped 1 previous similar message [ 3858.400513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3858.405469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.966499] LustreError: 132285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.978679] LustreError: 132285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3862.485302] Lustre: server umount lustre-OST0000 complete [ 3866.080727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.085488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.087031] Lustre: Skipped 1 previous similar message [ 3866.093481] Lustre: Skipped 4 previous similar messages [ 3870.495333] Lustre: server umount lustre-MDT0000 complete [ 3871.200444] LustreError: 131595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.206885] LustreError: 131595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3872.133262] LustreError: 130675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890385 with bad export cookie 7393001304014651914 [ 3872.137772] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3872.139071] LustreError: 130675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3872.294514] Lustre: server umount lustre-MDT0001 complete [ 3880.883211] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3882.213637] Key type lgssc unregistered [ 3882.409509] LNet: 136566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3882.414181] LNetError: 136566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3883.429643] LNet: Removed LNI 192.168.202.152@tcp [ 3883.877497] Key type .llcrypt unregistered [ 3883.879237] Key type ._llcrypt unregistered [ 3903.773710] Key type ._llcrypt registered [ 3903.775665] Key type .llcrypt registered [ 3903.845859] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3912.846349] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3913.591943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3913.600163] alg: No test for adler32 (adler32-zlib) [ 3914.531279] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 3914.660747] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3916.303377] Key type lgssc registered [ 3916.943409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3933.690697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3933.710540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.852401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3934.870336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3934.910908] Lustre: lustre-MDT0000: new disk, initializing [ 3934.948544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.957598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3936.648372] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3942.301774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.361211] Lustre: 140661:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3942.395620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3942.398861] Lustre: Skipped 1 previous similar message [ 3942.445187] Lustre: lustre-MDT0001: new disk, initializing [ 3942.492851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3942.518495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3942.524202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3944.376375] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3948.607902] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.462305] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3953.862745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3954.027566] Lustre: lustre-OST0000: new disk, initializing [ 3954.029655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3954.072627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.393024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3954.397101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3954.436937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3956.612499] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3960.228710] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3963.484547] Lustre: Failing over lustre-OST0000 [ 3963.572681] Lustre: server umount lustre-OST0000 complete [ 3977.455136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.603105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.621216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.636838] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3977.936397] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3980.424608] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3981.279089] Lustre: 138888:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890478/real 1774890478] req@000000004e33cc07 x1861107504409728/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774890494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3982.224526] Lustre: Failing over lustre-MDT0000 [ 3982.352817] Lustre: server umount lustre-MDT0000 complete [ 3982.821498] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.828547] Lustre: Skipped 3 previous similar messages [ 3985.887149] Lustre: 138889:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890483/real 1774890483] req@00000000fac018bf x1861107504410880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774890499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3985.899040] Lustre: 138889:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3987.936339] LustreError: 140672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.946354] LustreError: 140672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3991.007096] Lustre: 138886:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890488/real 1774890488] req@000000006712e224 x1861107504412032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774890504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3993.057049] LustreError: 140672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.063919] LustreError: 140672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3996.422264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.506690] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.515238] Lustre: 140663:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.202.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3996.612070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3996.638277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3996.643870] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3996.644248] Lustre: lustre-OST0000: Recovery over after 0:19, of 3 clients 3 recovered and 0 were evicted. [ 3996.667986] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3998.348808] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3998.405769] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4001.766383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4001.769323] Lustre: Skipped 1 previous similar message [ 4001.780585] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4001.804961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4006.883293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4006.886519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.887905] LustreError: Skipped 1 previous similar message [ 4006.893258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4006.894692] Lustre: Skipped 1 previous similar message [ 4006.900656] Lustre: Skipped 1 previous similar message [ 4009.764338] Lustre: server umount lustre-OST0000 complete [ 4012.000965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.001053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4012.002347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4012.012809] Lustre: Skipped 1 previous similar message [ 4017.120583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4017.123568] Lustre: Skipped 2 previous similar messages [ 4017.772741] Lustre: server umount lustre-MDT0000 complete [ 4019.725217] LustreError: 140652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890532 with bad export cookie 11922543474831102793 [ 4019.728640] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.731733] LustreError: 140652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4019.909804] Lustre: server umount lustre-MDT0001 complete [ 4024.718288] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4026.352749] Key type lgssc unregistered [ 4026.528584] LNet: 145886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4026.534467] LNetError: 145886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4026.545571] LNet: Removed LNI 192.168.202.152@tcp [ 4027.042692] Key type .llcrypt unregistered [ 4027.045605] Key type ._llcrypt unregistered [ 4050.209383] Key type ._llcrypt registered [ 4050.211118] Key type .llcrypt registered [ 4050.258674] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4050.861718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4050.872818] alg: No test for adler32 (adler32-zlib) [ 4051.784507] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 4051.881517] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4053.471161] Key type lgssc registered [ 4053.919641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4057.768691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4057.779310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.005591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.857707] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4064.711637] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4067.018138] LustreError: 147433:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4067.023830] LustreError: 147433:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4067.028494] LustreError: 147433:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 4067.148474] Lustre: server umount lustre-MDT0000 complete [ 4073.326264] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4077.601583] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4092.682951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.804259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4092.820346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4092.862172] Lustre: lustre-MDT0000: new disk, initializing [ 4092.899256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.909807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4094.517224] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4099.809798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.873834] Lustre: 150584:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4099.916655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4099.919199] Lustre: Skipped 1 previous similar message [ 4099.960243] Lustre: lustre-MDT0001: new disk, initializing [ 4100.001923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4100.018165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4100.022376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4101.769466] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4105.228046] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4106.644848] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4109.192107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.308252] Lustre: lustre-OST0000: new disk, initializing [ 4109.310821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4109.354172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.396178] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4114.435066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4114.441450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4114.458104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4115.376943] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4118.549343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4119.279055] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:10:31 (1774890631) [ 4228.064117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4228.067302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.072028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4229.599777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4229.599969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.602173] Lustre: Skipped 1 previous similar message [ 4229.607159] Lustre: Skipped 1 previous similar message [ 4234.719489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.722038] Lustre: Skipped 3 previous similar messages [ 4239.840783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.844033] Lustre: Skipped 2 previous similar messages [ 4240.351508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4240.436449] Lustre: server umount lustre-MDT0000 complete [ 4241.977721] LustreError: 151361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890755 with bad export cookie 1782197843628183447 [ 4241.977721] LustreError: 150575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890755 with bad export cookie 1782197843628183447 [ 4241.977856] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.983846] LustreError: 150575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4244.960575] LustreError: 150592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.960850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.970188] LustreError: 150592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4244.978339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4247.007408] LustreError: 150592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.183285] Lustre: server umount lustre-MDT0001 complete [ 4255.863619] Lustre: server umount lustre-OST0000 complete [ 4259.863170] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4263.913809] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4278.072983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.181669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4278.196792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4278.233289] Lustre: lustre-MDT0000: new disk, initializing [ 4278.257843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.264732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4279.667557] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4284.593775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.640701] Lustre: 156810:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4284.673666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4284.675589] Lustre: Skipped 1 previous similar message [ 4284.717491] Lustre: lustre-MDT0001: new disk, initializing [ 4284.781474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.787557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4286.541865] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4290.334984] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4291.957759] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4295.005665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4295.141329] Lustre: lustre-OST0000: new disk, initializing [ 4295.144594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4295.199310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.204251] Lustre: Skipped 1 previous similar message [ 4297.575382] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4301.175630] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4301.312276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4301.315933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4301.361933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4302.862727] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4302.959767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4304.795684] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4304.909289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4306.401386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.402381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4306.406595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4306.408255] LustreError: Skipped 1 previous similar message [ 4306.415748] Lustre: Skipped 2 previous similar messages [ 4307.936599] Lustre: server umount lustre-OST0000 complete [ 4310.496749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4310.502416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.514690] Lustre: Skipped 1 previous similar message [ 4324.319132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4324.429136] Lustre: server umount lustre-MDT0000 complete [ 4325.858553] LustreError: 156821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.866372] LustreError: 156821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4326.267349] LustreError: 156800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890839 with bad export cookie 1782197843628342242 [ 4326.269219] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.272210] LustreError: 156800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4326.429319] Lustre: server umount lustre-MDT0001 complete [ 4330.359707] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:14:02 (1774890842) [ 4338.025364] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4341.480663] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4355.433453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.554713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4355.572565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4355.614426] Lustre: lustre-MDT0000: new disk, initializing [ 4355.650873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.658808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4357.088904] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4361.801685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.860971] Lustre: 163117:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4361.977919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4363.491218] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4366.756104] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4368.115146] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4370.877670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.407829] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4375.560705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4375.621796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4376.988474] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4380.643580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4380.647796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.652815] Lustre: Skipped 1 previous similar message [ 4380.655835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4380.657952] Lustre: Skipped 8 previous similar messages [ 4381.287330] Lustre: server umount lustre-OST0000 complete [ 4384.860152] Lustre: server umount lustre-MDT0000 complete [ 4385.761976] LustreError: 164265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.768597] LustreError: 164265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4386.270498] LustreError: 163900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890899 with bad export cookie 1782197843628343110 [ 4386.271233] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4386.275287] LustreError: 163900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4386.394120] Lustre: server umount lustre-MDT0001 complete [ 4389.732461] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:15:01 (1774890901) [ 4396.135838] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4399.906149] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4414.707508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.830584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4414.845116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4414.851389] Lustre: Skipped 3 previous similar messages [ 4414.894658] Lustre: lustre-MDT0000: new disk, initializing [ 4414.896081] Lustre: Skipped 2 previous similar messages [ 4414.938841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.942947] Lustre: Skipped 2 previous similar messages [ 4414.953674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4414.958448] Lustre: Skipped 2 previous similar messages [ 4416.753429] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4421.568202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.617390] Lustre: 169033:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4421.755755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.315639] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4426.885272] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4428.422929] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4431.217762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4433.576346] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4434.611462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4437.424795] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4440.032215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4440.034851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.037973] LustreError: Skipped 2 previous similar messages [ 4440.043927] Lustre: Skipped 3 previous similar messages [ 4440.049658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4440.055139] Lustre: Skipped 1 previous similar message [ 4445.923258] Lustre: server umount lustre-OST0000 complete [ 4453.667369] Lustre: server umount lustre-MDT0000 complete [ 4455.082984] LustreError: 169816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890968 with bad export cookie 1782197843628344181 [ 4455.085893] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.088257] LustreError: 169816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.227754] Lustre: server umount lustre-MDT0001 complete [ 4459.222490] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4460.395762] Key type lgssc unregistered [ 4460.545536] LNet: 172507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4460.549121] LNetError: 172507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4460.559525] LNet: Removed LNI 192.168.202.152@tcp [ 4460.909054] Key type .llcrypt unregistered [ 4460.910749] Key type ._llcrypt unregistered [ 4471.201759] Key type ._llcrypt registered [ 4471.203295] Key type .llcrypt registered [ 4471.262840] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4471.715256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4471.764346] alg: No test for adler32 (adler32-zlib) [ 4472.654600] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 4472.753910] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4474.343182] Key type lgssc registered [ 4474.819564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4477.941418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4477.949656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.135739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.617772] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4483.877480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.041376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.523683] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4487.844883] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4489.374829] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4492.293676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.423295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.548933] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4496.697232] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4504.741762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4520.416592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4520.421135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.422406] LustreError: Skipped 1 previous similar message [ 4520.430552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4520.434669] Lustre: Skipped 1 previous similar message [ 4525.536513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4525.539366] Lustre: Skipped 1 previous similar message [ 4526.171520] Lustre: server umount lustre-OST0000 complete [ 4530.144620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4530.148597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.154418] Lustre: Skipped 1 previous similar message [ 4530.156966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4533.851704] Lustre: server umount lustre-MDT0000 complete [ 4535.384822] LustreError: 173864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891048 with bad export cookie 10223213651763157538 [ 4535.387673] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4535.389354] LustreError: 173864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4535.514315] Lustre: server umount lustre-MDT0001 complete [ 4539.541218] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4540.639910] Key type lgssc unregistered [ 4540.775945] LNet: 177084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4540.780710] LNetError: 177084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4540.791388] LNet: Removed LNI 192.168.202.152@tcp [ 4541.146159] Key type .llcrypt unregistered [ 4541.147759] Key type ._llcrypt unregistered [ 4551.109797] Key type ._llcrypt registered [ 4551.111647] Key type .llcrypt registered [ 4551.177236] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4551.527970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4551.537290] alg: No test for adler32 (adler32-zlib) [ 4552.409951] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 4552.522037] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4554.111510] Key type lgssc registered [ 4554.538810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4557.723166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4557.732403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.929402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.435070] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4563.914172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.118595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.724142] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4568.043176] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4569.377983] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4572.121435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.257438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4574.412820] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4576.453277] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4584.553353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4590.048538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4590.050156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.051050] LustreError: Skipped 1 previous similar message [ 4590.052027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4590.054348] Lustre: Skipped 1 previous similar message [ 4592.333868] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 4592.337239] Lustre: Skipped 1 previous similar message [ 4595.172225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.178800] Lustre: Skipped 1 previous similar message [ 4595.538191] Lustre: server umount lustre-OST0000 complete [ 4597.758446] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ca9c5db0000.ost_server_uuid 50 [ 4598.440697] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4610.528120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4610.529106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.531822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4610.536374] Lustre: Skipped 1 previous similar message [ 4610.541609] Lustre: Skipped 1 previous similar message [ 4614.873412] Lustre: server umount lustre-MDT0000 complete [ 4615.648175] LustreError: 179679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.653653] LustreError: 179679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4616.299539] LustreError: 178440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891129 with bad export cookie 11542145156191281368 [ 4616.300178] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.306173] LustreError: 178440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4616.425649] Lustre: server umount lustre-MDT0001 complete [ 4619.737949] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:18:51 (1774891131) [ 4622.347028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.516370] LustreError: 181943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.541250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.963207] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4626.975372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.130451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.527997] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4630.640980] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4632.055648] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4634.686837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4636.935931] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4639.110615] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4641.980241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.062264] Lustre: lustre-OST0001: new disk, initializing [ 4642.065525] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4642.108479] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4642.110826] Lustre: Skipped 1 previous similar message [ 4642.319148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4642.326924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4642.398555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4644.473594] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4647.460701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4648.027601] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4652.515354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.516643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4652.521546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4652.523310] LustreError: Skipped 1 previous similar message [ 4655.248605] Lustre: server umount lustre-OST0000 complete [ 4657.468871] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4657.560884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4657.635213] LustreError: 183541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.643312] LustreError: 183541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4659.132824] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4659.225353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4662.754351] LustreError: 183544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.761320] LustreError: 183544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4667.874398] LustreError: 183541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.881680] LustreError: 183541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4672.993673] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4672.994307] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4672.996665] LustreError: Skipped 1 previous similar message [ 4673.003889] Lustre: Skipped 1 previous similar message [ 4673.006694] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4673.012989] Lustre: Skipped 3 previous similar messages [ 4675.408571] Lustre: server umount lustre-OST0001 complete [ 4678.113065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.036039] Lustre: server umount lustre-MDT0000 complete [ 4683.232189] LustreError: 181943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.238994] LustreError: 181943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4684.440158] LustreError: 181923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891197 with bad export cookie 11542145156191282425 [ 4684.443646] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.445182] LustreError: 181923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4684.572737] Lustre: server umount lustre-MDT0001 complete [ 4693.696598] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:20:05 (1774891205) [ 4696.305993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.419947] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4696.433064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4696.514239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.996790] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4701.203598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4701.250947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4701.266398] Lustre: 187721:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4702.846991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4704.864668] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4706.253965] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4709.008215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.084143] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4711.432845] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4713.428344] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4715.956769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.990336] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4718.152074] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4719.142057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4719.148638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4720.143983] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4724.194170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4724.196446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.197709] LustreError: Skipped 1 previous similar message [ 4724.198329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.202915] Lustre: Skipped 2 previous similar messages [ 4724.204495] Lustre: Skipped 2 previous similar messages [ 4727.287860] Lustre: server umount lustre-OST0000 complete [ 4729.315730] LustreError: 189343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.320448] LustreError: 189343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4739.557128] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4739.561094] LustreError: Skipped 1 previous similar message [ 4744.217117] Lustre: server umount lustre-OST0001 complete [ 4747.743927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.747469] Lustre: Skipped 2 previous similar messages [ 4751.886949] Lustre: server umount lustre-MDT0000 complete [ 4753.285085] LustreError: 187713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891266 with bad export cookie 11542145156191283783 [ 4753.287610] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4753.289266] LustreError: 187713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4753.416738] Lustre: server umount lustre-MDT0001 complete [ 4762.218902] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:21:14 (1774891274) [ 4767.923474] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4771.361959] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4785.741223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.837576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4785.858519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4785.895880] Lustre: lustre-MDT0000: new disk, initializing [ 4785.928651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.931505] Lustre: Skipped 3 previous similar messages [ 4785.938288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4787.568510] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4792.204260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.246543] Lustre: 195289:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4792.269529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4792.272033] Lustre: Skipped 1 previous similar message [ 4792.306955] Lustre: lustre-MDT0001: new disk, initializing [ 4792.355276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4792.358806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4793.800407] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4797.169303] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4798.607788] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4801.313956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.420441] Lustre: lustre-OST0000: new disk, initializing [ 4801.423495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4801.768525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4801.771937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4801.812255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4803.588819] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4807.019104] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4808.539912] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4808.624260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4810.137130] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4810.222510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4812.258425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.258524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.259300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.259305] Lustre: Skipped 6 previous similar messages [ 4812.261740] LustreError: Skipped 2 previous similar messages [ 4812.266195] Lustre: Skipped 2 previous similar messages [ 4817.048199] Lustre: server umount lustre-OST0000 complete [ 4824.615487] Lustre: server umount lustre-MDT0000 complete [ 4826.014305] LustreError: 195325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891339 with bad export cookie 11542145156191285456 [ 4826.016545] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4826.024401] LustreError: 195325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4826.157728] Lustre: server umount lustre-MDT0001 complete [ 4829.427112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.608219] LustreError: 199098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.614857] LustreError: 199098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4831.012333] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4834.139550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.746860] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4837.660399] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4838.994155] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4841.513939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4843.570304] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4845.510762] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4846.913330] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4854.143479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4855.626318] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4855.697465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4859.364476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4862.480829] Lustre: server umount lustre-OST0000 complete [ 4864.541608] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4866.066191] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4890.118304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4890.259269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.264382] Lustre: Skipped 5 previous similar messages [ 4892.294384] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4894.288707] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4894.978424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4898.129278] Lustre: server umount lustre-OST0000 complete [ 4900.832512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.838259] Lustre: Skipped 4 previous similar messages [ 4900.840800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.843064] Lustre: Skipped 5 previous similar messages [ 4901.664305] Lustre: server umount lustre-MDT0000 complete [ 4903.041161] LustreError: 199079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891416 with bad export cookie 11542145156191286324 [ 4903.044958] LustreError: 199079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4903.045069] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.159170] Lustre: server umount lustre-MDT0001 complete [ 4906.240664] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:23:38 (1774891418) [ 4908.479707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.641739] LustreError: 204592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.080263] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4913.053943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4914.627821] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4916.633487] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.974436] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4920.413400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.561726] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4924.554031] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4926.033711] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4934.309982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4935.791547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4935.881071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4938.378893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4938.467684] Lustre: lustre-OST0001: new disk, initializing [ 4938.469993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4940.641058] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4944.165602] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4945.683114] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4947.967216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4947.970744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4947.997751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4948.819366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4950.241902] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4950.320794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4953.058913] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4953.061471] LustreError: Skipped 1 previous similar message [ 4957.122764] Lustre: server umount lustre-OST0001 complete [ 4959.388547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4959.482356] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4960.973733] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4961.053665] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4984.786222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.875939] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4988.779467] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4989.342069] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4989.929875] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4989.934657] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4989.934693] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4996.087766] Lustre: server umount lustre-OST0001 complete [ 5014.357297] Lustre: server umount lustre-OST0000 complete [ 5029.856777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.858840] Lustre: Skipped 12 previous similar messages [ 5030.367130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5030.429551] Lustre: server umount lustre-MDT0000 complete [ 5031.742334] LustreError: 204572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891544 with bad export cookie 11542145156191287248 [ 5031.745637] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.746014] LustreError: 204572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5046.243446] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5046.341674] Lustre: server umount lustre-MDT0001 complete [ 5055.165152] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:26:07 (1774891567) [ 5057.523026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.628388] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5057.641935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5057.728928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.732184] Lustre: Skipped 5 previous similar messages [ 5059.304598] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5062.742534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5062.782292] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5062.794781] Lustre: 212609:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5064.470805] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5066.519601] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5067.848512] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5070.440226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5070.511866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5072.670365] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5074.702903] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5084.053517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5084.080175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5084.113094] LustreError: 214229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.120975] LustreError: 214229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 5086.252393] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5088.487482] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5090.199825] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5091.858974] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5091.953591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5093.454848] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5093.533259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5094.956936] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ca9c8d5b800.ost_server_uuid 50 [ 5095.593948] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ca9c8d5b800.ost_server_uuid in FULL state after 0 sec [ 5096.031469] Lustre: Permanently deactivating lustre-OST0001 [ 5096.043846] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5097.072296] Lustre: Permanently reactivating lustre-OST0001 [ 5099.493460] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5099.493586] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.496862] LustreError: Skipped 5 previous similar messages [ 5099.500864] Lustre: Skipped 8 previous similar messages [ 5104.253529] Lustre: server umount lustre-OST0001 complete [ 5111.965455] Lustre: server umount lustre-OST0000 complete [ 5119.594626] Lustre: server umount lustre-MDT0000 complete [ 5121.021663] LustreError: 214244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891634 with bad export cookie 11542145156191288529 [ 5121.025342] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.026652] LustreError: 214244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5121.151795] Lustre: server umount lustre-MDT0001 complete [ 5130.343707] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:27:22 (1774891642) [ 5133.314942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5133.432267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5133.443649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5133.446533] Lustre: Skipped 5 previous similar messages [ 5135.134272] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5138.564625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.611461] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5138.624164] Lustre: 218714:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5140.309552] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5142.440242] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5143.819767] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5146.526245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5146.621248] Lustre: Permanently deactivating lustre-OST0000 [ 5146.626684] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5148.850062] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5150.068232] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5150.072905] LustreError: 220824:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 5150.914107] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5153.827145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.146465] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5158.185382] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5159.981754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5164.165874] Lustre: Permanently reactivating lustre-OST0000 [ 5164.172757] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5164.175086] Lustre: Skipped 2 previous similar messages [ 5169.681880] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5169.685642] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.694264] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5169.698051] Lustre: Skipped 1 previous similar message [ 5169.710411] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5169.722976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5177.471645] Lustre: server umount lustre-OST0001 complete [ 5181.017858] Lustre: server umount lustre-OST0000 complete [ 5188.640623] Lustre: server umount lustre-MDT0000 complete [ 5190.002340] LustreError: 218705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891703 with bad export cookie 11542145156191290188 [ 5190.006259] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5190.007224] LustreError: 218705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5190.137665] Lustre: server umount lustre-MDT0001 complete [ 5194.223282] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5195.385959] Key type lgssc unregistered [ 5195.531424] LNet: 223096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5195.534107] LNetError: 223096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5195.543316] LNet: Removed LNI 192.168.202.152@tcp [ 5195.880673] Key type .llcrypt unregistered [ 5195.882212] Key type ._llcrypt unregistered [ 5205.775842] Key type ._llcrypt registered [ 5205.776925] Key type .llcrypt registered [ 5205.840576] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5206.175672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5206.186023] alg: No test for adler32 (adler32-zlib) [ 5207.079902] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5207.175429] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5208.775186] Key type lgssc registered [ 5209.202631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5213.263195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5213.271912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.475604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5215.923042] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5219.312588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.358612] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5219.370359] Lustre: 224513:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5219.374786] Lustre: Permanently deactivating lustre-MDT0001 [ 5219.381279] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5219.515436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.946482] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5223.193231] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5223.282862] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5223.285128] LustreError: 224555:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5223.293697] LustreError: 224555:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5223.296887] LustreError: 224555:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 5224.694437] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5227.482398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5227.619494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.672946] LustreError: 226138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.678907] LustreError: 226138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5229.973943] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5232.237168] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5234.997925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5235.095220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.226116] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5239.210941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5239.213763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5239.213965] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5239.276641] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5240.902241] Lustre: Permanently reactivating lustre-MDT0001 [ 5240.908482] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5240.910382] Lustre: Skipped 2 previous similar messages [ 5245.647670] Lustre: lustre-MDT0001: Client 118249c0-ca04-467d-a8b8-b1c28a92474a (at 192.168.202.52@tcp) reconnecting [ 5247.356629] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5247.965767] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5249.774449] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5249.850489] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5251.289158] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5251.368114] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5252.236209] Lustre: Permanently deactivating lustre-MDT0001 [ 5252.242189] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5252.244667] Lustre: Skipped 2 previous similar messages [ 5252.296950] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5260.768217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5260.771146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.776383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.794819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.796159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.802022] Lustre: Skipped 2 previous similar messages [ 5261.805841] Lustre: Skipped 3 previous similar messages [ 5262.941650] Lustre: server umount lustre-MDT0000 complete [ 5264.293784] LustreError: 224506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891777 with bad export cookie 6007817269431954390 [ 5264.296774] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5264.297924] LustreError: 224506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5264.413307] Lustre: server umount lustre-MDT0001 complete [ 5271.874949] Lustre: server umount lustre-OST0000 complete [ 5275.980521] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:29:48 (1774891788) [ 5280.580839] Lustre: server umount lustre-OST0001 complete [ 5283.525562] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5286.737213] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5300.048631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.146454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5300.150852] Lustre: Skipped 2 previous similar messages [ 5300.161994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5300.200606] Lustre: lustre-MDT0000: new disk, initializing [ 5300.232677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5300.241675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5301.678533] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5306.178855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.223752] Lustre: 232163:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5306.249320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5306.251660] Lustre: Skipped 1 previous similar message [ 5306.287798] Lustre: lustre-MDT0001: new disk, initializing [ 5306.335523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5306.339534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5307.790739] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5311.077403] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5312.444914] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5315.049318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5315.173618] Lustre: lustre-OST0000: new disk, initializing [ 5315.176546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5315.213911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.216640] Lustre: Skipped 1 previous similar message [ 5317.181745] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5317.304705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5317.307494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5317.341177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5320.323471] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5321.024854] LustreError: 233452:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5325.388686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5325.455117] Lustre: lustre-OST0001: new disk, initializing [ 5325.457445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5327.563547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5330.919389] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5331.119111] LustreError: 233452:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5331.128591] LustreError: 233452:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5331.557701] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5332.995341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5332.999859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5333.042837] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5341.223360] LustreError: 233452:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5343.201021] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.202112] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.204815] LustreError: Skipped 1 previous similar message [ 5343.210208] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5343.213986] Lustre: Skipped 1 previous similar message [ 5348.037584] Lustre: server umount lustre-OST0001 complete [ 5348.324855] LustreError: 233888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.329517] LustreError: 233888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5353.443496] LustreError: 233887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.448549] LustreError: 233887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5358.565547] LustreError: 233888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.571326] LustreError: 233888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5363.680109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.681506] LustreError: 233887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5363.683374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5363.684619] Lustre: Skipped 2 previous similar messages [ 5363.685850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.691581] LustreError: 233887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5363.700784] Lustre: Skipped 1 previous similar message [ 5366.229370] Lustre: server umount lustre-OST0000 complete [ 5367.776506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5367.780580] LustreError: Skipped 1 previous similar message [ 5367.782407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.787795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.789428] Lustre: server umount lustre-MDT0000 complete [ 5373.920018] LustreError: 232173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.927235] LustreError: 232173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5375.095325] LustreError: 234907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891888 with bad export cookie 6007817269431956980 [ 5375.098425] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.099747] LustreError: 234907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5381.310107] Lustre: server umount lustre-MDT0001 complete [ 5385.215624] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5386.329191] Key type lgssc unregistered [ 5386.465925] LNet: 236806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5386.468951] LNetError: 236806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.479357] LNet: Removed LNI 192.168.202.152@tcp [ 5386.796907] Key type .llcrypt unregistered [ 5386.798150] Key type ._llcrypt unregistered [ 5402.118418] Key type ._llcrypt registered [ 5402.119688] Key type .llcrypt registered [ 5402.167609] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5402.544192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5402.598849] alg: No test for adler32 (adler32-zlib) [ 5403.498909] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5403.592692] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5405.191137] Key type lgssc registered [ 5405.596841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.534395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5408.543896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.665729] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5409.673620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5409.733768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.088475] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5414.442465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.516209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5414.528704] Lustre: 238973:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5414.636835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5416.096211] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5418.190517] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5419.568467] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5422.045485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5422.122635] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5422.196773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.489446] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5426.432137] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5426.706838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5472.737510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.740220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.747439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5472.750976] Lustre: Skipped 1 previous similar message [ 5477.423685] Lustre: server umount lustre-OST0000 complete [ 5479.068183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5483.560921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.676279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5485.741055] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5487.844702] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5492.901832] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.907148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.907280] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5492.917485] Lustre: Skipped 1 previous similar message [ 5523.938456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.939053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5523.940563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5523.944163] Lustre: Skipped 2 previous similar messages [ 5523.957630] LustreError: Skipped 1 previous similar message [ 5528.106987] Lustre: server umount lustre-OST0000 complete [ 5532.643155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5532.647314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5532.652777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.654954] Lustre: Skipped 1 previous similar message [ 5535.773981] Lustre: server umount lustre-MDT0000 complete [ 5537.144459] LustreError: 239697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892050 with bad export cookie 299133073122450943 [ 5537.148575] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5537.150243] LustreError: 239697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5537.281330] Lustre: server umount lustre-MDT0001 complete [ 5541.448255] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5542.604800] Key type lgssc unregistered [ 5542.755524] LNet: 243925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5542.758656] LNetError: 243925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5542.765276] LNet: Removed LNI 192.168.202.152@tcp [ 5543.165103] Key type .llcrypt unregistered [ 5543.166716] Key type ._llcrypt unregistered [ 5553.703112] Key type ._llcrypt registered [ 5553.704188] Key type .llcrypt registered [ 5553.756350] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5554.129968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5554.196706] alg: No test for adler32 (adler32-zlib) [ 5555.092951] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5555.211453] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5556.815180] Key type lgssc registered [ 5557.296709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5560.500498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5560.508845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.733638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.364362] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5566.679482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.838179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5568.493147] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5570.861831] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.328467] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5574.944237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.083811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.311463] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5579.519575] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.345475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5601.763689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.763747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.765728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.770218] Lustre: Skipped 1 previous similar message [ 5605.722336] Lustre: server umount lustre-OST0000 complete [ 5607.904320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5607.906959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.912934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5607.916197] Lustre: Skipped 1 previous similar message [ 5612.003871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.003830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.471391] Lustre: server umount lustre-MDT0000 complete [ 5614.864446] LustreError: 245284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892128 with bad export cookie 7222326815081879020 [ 5614.867608] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.869359] LustreError: 245284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5615.007900] Lustre: server umount lustre-MDT0001 complete [ 5618.806408] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5619.875130] Key type lgssc unregistered [ 5620.019440] LNet: 249271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5620.021812] LNetError: 249271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5620.033306] LNet: Removed LNI 192.168.202.152@tcp [ 5620.370370] Key type .llcrypt unregistered [ 5620.371835] Key type ._llcrypt unregistered [ 5627.724955] Key type ._llcrypt registered [ 5627.726754] Key type .llcrypt registered [ 5627.775191] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5628.199663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5628.218636] alg: No test for adler32 (adler32-zlib) [ 5629.075543] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5629.184384] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5630.767197] Key type lgssc registered [ 5631.191069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.047819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.053208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.212556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.608638] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5639.737223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.863770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5641.254314] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5643.188490] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5644.389239] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5646.780812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5646.911841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.967911] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5651.024483] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5660.265644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5665.762985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5665.763573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.764505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5665.766357] LustreError: Skipped 1 previous similar message [ 5665.769904] Lustre: Skipped 1 previous similar message [ 5667.112460] Lustre: server umount lustre-OST0000 complete [ 5670.681320] Lustre: server umount lustre-MDT0000 complete [ 5670.881632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5670.881755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.893929] LustreError: 251162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.906086] LustreError: 251162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5672.201178] LustreError: 250434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892185 with bad export cookie 16709567396748486735 [ 5672.204290] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.205906] LustreError: 250434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5672.337650] Lustre: server umount lustre-MDT0001 complete [ 5676.160055] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5677.327608] Key type lgssc unregistered [ 5677.461333] LNet: 253794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5677.463889] LNetError: 253794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5677.476601] LNet: Removed LNI 192.168.202.152@tcp [ 5677.807349] Key type .llcrypt unregistered [ 5677.808471] Key type ._llcrypt unregistered [ 5687.411030] Key type ._llcrypt registered [ 5687.411879] Key type .llcrypt registered [ 5687.454593] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5687.883579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5687.890532] alg: No test for adler32 (adler32-zlib) [ 5688.754396] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5688.835332] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5690.423637] Key type lgssc registered [ 5690.842957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5693.722303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5693.729101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.894860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.346752] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5699.659916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.821242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.462615] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5703.595616] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5704.939463] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5707.509566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5707.663151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.833620] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5711.923397] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5719.976236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5731.296147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5731.299258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.304766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5732.756238] Lustre: server umount lustre-OST0000 complete [ 5735.392662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.396792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5735.398760] Lustre: Skipped 1 previous similar message [ 5740.512258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5740.514416] Lustre: Skipped 1 previous similar message [ 5744.648990] Lustre: server umount lustre-MDT0000 complete [ 5745.632068] LustreError: 255169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.638103] LustreError: 255169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5746.064377] LustreError: 258117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892259 with bad export cookie 66938677193649141 [ 5746.067207] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.068534] LustreError: 258117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5746.197020] Lustre: server umount lustre-MDT0001 complete [ 5749.871785] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5750.928628] Key type lgssc unregistered [ 5751.088629] LNet: 259138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5751.092303] LNetError: 259138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5751.108445] LNet: Removed LNI 192.168.202.152@tcp [ 5751.465400] Key type .llcrypt unregistered [ 5751.466693] Key type ._llcrypt unregistered [ 5758.828683] Key type ._llcrypt registered [ 5758.831084] Key type .llcrypt registered [ 5758.881940] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5759.309307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5759.336463] alg: No test for adler32 (adler32-zlib) [ 5760.210524] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5760.313634] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5761.903204] Key type lgssc registered [ 5762.369789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5765.400424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5765.412868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.598748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5768.005649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5770.941166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.061521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.383119] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5774.196068] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5775.404231] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5777.677515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.792676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.746611] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5781.656117] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5791.145857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5796.322341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5796.324664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.324696] LustreError: Skipped 1 previous similar message [ 5796.330247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5796.333170] Lustre: Skipped 1 previous similar message [ 5797.925673] Lustre: server umount lustre-OST0000 complete [ 5801.443341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.454279] Lustre: Skipped 1 previous similar message [ 5801.459338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5801.462073] Lustre: Skipped 1 previous similar message [ 5801.629116] Lustre: server umount lustre-MDT0000 complete [ 5803.086038] LustreError: 260300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892316 with bad export cookie 9127192432971560065 [ 5803.086513] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5803.092776] LustreError: 260300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5803.227958] Lustre: server umount lustre-MDT0001 complete [ 5807.160126] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5808.410408] Key type lgssc unregistered [ 5808.570335] LNet: 263698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5808.573692] LNetError: 263698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5808.586364] LNet: Removed LNI 192.168.202.152@tcp [ 5808.906275] Key type .llcrypt unregistered [ 5808.907441] Key type ._llcrypt unregistered [ 5813.164259] Key type ._llcrypt registered [ 5813.165942] Key type .llcrypt registered [ 5813.217991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5819.644156] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5826.023724] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5826.396272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5826.436776] alg: No test for adler32 (adler32-zlib) [ 5827.291777] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5827.375655] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5828.959224] Key type lgssc registered [ 5829.361327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5842.262797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5842.271970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.382558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5843.394760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5843.437703] Lustre: lustre-MDT0000: new disk, initializing [ 5843.462970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.474306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5844.982410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5849.617726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.678770] Lustre: 267177:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5849.710770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5849.712821] Lustre: Skipped 1 previous similar message [ 5849.756466] Lustre: lustre-MDT0001: new disk, initializing [ 5849.806609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.822213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5849.828412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5851.494353] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5854.744538] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5855.988798] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5858.552913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5858.680811] Lustre: lustre-OST0000: new disk, initializing [ 5858.683316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5858.717308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.922028] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5863.432227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5863.440704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5863.499158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5864.236276] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5865.659501] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5865.748599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5867.107505] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5867.182031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5868.513697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.517407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.519561] LustreError: Skipped 1 previous similar message [ 5868.520122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.523876] Lustre: Skipped 1 previous similar message [ 5869.970683] Lustre: server umount lustre-OST0000 complete [ 5873.632972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.633937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.638723] Lustre: Skipped 1 previous similar message [ 5873.644788] Lustre: Skipped 2 previous similar messages [ 5877.606794] Lustre: server umount lustre-MDT0000 complete [ 5879.116628] LustreError: 267170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892392 with bad export cookie 10495147594689800873 [ 5879.118356] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5879.128797] LustreError: 267170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5879.272365] Lustre: server umount lustre-MDT0001 complete [ 5882.477203] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:39:54 (1774892394) [ 5884.900459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.053673] LustreError: 271176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.060021] LustreError: 271176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5885.078681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5886.494125] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5889.643585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.820951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5891.514524] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5893.610465] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5894.873777] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5897.407749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5899.671755] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5901.726375] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5916.128239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.132991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.142056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.224903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.226949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.229574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5921.363831] Lustre: server umount lustre-OST0000 complete [ 5925.344761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.349958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.350017] Lustre: Skipped 1 previous similar message [ 5925.352679] Lustre: Skipped 1 previous similar message [ 5928.803490] Lustre: server umount lustre-MDT0000 complete [ 5930.186155] LustreError: 271155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892443 with bad export cookie 10495147594689801741 [ 5930.188655] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5930.188963] LustreError: 271895:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5930.190724] LustreError: 271155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5930.320171] Lustre: server umount lustre-MDT0001 complete [ 5934.327591] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5935.418087] Key type lgssc unregistered [ 5935.580279] LNet: 274374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.583761] LNetError: 274374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.592458] LNet: Removed LNI 192.168.202.152@tcp [ 5935.901847] Key type .llcrypt unregistered [ 5935.903291] Key type ._llcrypt unregistered [ 5946.126564] Key type ._llcrypt registered [ 5946.127783] Key type .llcrypt registered [ 5946.163649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5952.318456] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5952.650792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5952.658757] alg: No test for adler32 (adler32-zlib) [ 5953.520661] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5953.617088] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5955.200142] Key type lgssc registered [ 5955.629557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5971.678377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5971.698544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.794607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5972.811890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5972.851874] Lustre: lustre-MDT0000: new disk, initializing [ 5972.882850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.890443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5974.346014] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5978.983848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.027941] Lustre: 278136:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5979.058252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5979.060883] Lustre: Skipped 1 previous similar message [ 5979.098405] Lustre: lustre-MDT0001: new disk, initializing [ 5979.143591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5979.163812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5979.169569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5980.610573] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5983.998635] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5985.528659] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5988.025375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.139063] Lustre: lustre-OST2710: new disk, initializing [ 5988.141464] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5988.176747] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5990.406992] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5993.768911] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5994.491346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5994.494198] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5994.548215] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5996.297688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.366712] Lustre: lustre-OST03e8: new disk, initializing [ 5996.369543] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5996.419639] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5998.410846] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6000.434095] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6001.672133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6001.675835] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6001.711991] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6004.263279] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6004.358707] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6005.771518] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6005.851944] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6007.391744] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6007.465751] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6008.803727] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6008.875852] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6011.875127] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6011.876886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6011.879405] Lustre: Skipped 1 previous similar message [ 6011.885167] Lustre: Skipped 3 previous similar messages [ 6016.006716] LustreError: 282463:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000461acfdb) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6018.032815] Lustre: server umount lustre-MDT0000 complete [ 6019.245075] LustreError: 278129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892532 with bad export cookie 1664213907435167935 [ 6019.248943] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6019.249523] LustreError: 278129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6019.366938] Lustre: server umount lustre-MDT0001 complete [ 6030.339198] Lustre: server umount lustre-OST2710 complete [ 6041.836496] Lustre: server umount lustre-OST03e8 complete [ 6048.202876] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6051.418639] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6063.854755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.938214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6063.949709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6063.984068] Lustre: lustre-MDT0000: new disk, initializing [ 6064.008293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.015364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6065.338848] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6069.758527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.808806] Lustre: 285851:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6069.936356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6071.348026] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6074.752189] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6076.072240] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6078.797767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6078.941181] Lustre: lustre-OST0000: new disk, initializing [ 6078.942797] Lustre: Skipped 1 previous similar message [ 6078.945133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6078.947066] Lustre: Skipped 2 previous similar messages [ 6078.977448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6078.981865] Lustre: Skipped 1 previous similar message [ 6081.191188] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6084.549817] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6085.112629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6085.115765] Lustre: Skipped 1 previous similar message [ 6085.117782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6085.150112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6086.112267] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6086.206161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6087.758659] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6087.845260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6090.210530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.210841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.211543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.215395] Lustre: Skipped 3 previous similar messages [ 6090.217885] LustreError: Skipped 1 previous similar message [ 6094.688292] Lustre: server umount lustre-OST0000 complete [ 6100.448721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.450291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.453207] Lustre: Skipped 1 previous similar message [ 6100.454973] Lustre: Skipped 1 previous similar message [ 6102.240301] Lustre: server umount lustre-MDT0000 complete [ 6103.676053] LustreError: 286637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892616 with bad export cookie 1664213907435169601 [ 6103.680145] LustreError: 286637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6103.681265] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6103.792627] Lustre: server umount lustre-MDT0001 complete [ 6106.748336] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:43:38 (1774892618) [ 6107.272370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6107.900241] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:43:39 (1774892619) [ 6108.942975] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6118.898419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.018093] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6119.026506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6119.095884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.568237] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6123.655113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.714741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6123.728616] Lustre: 291277:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6125.226106] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6127.150192] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6128.443183] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6141.827368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6141.914901] Lustre: Denying initial registration attempt from nid 192.168.202.152@tcp, specified as failover [ 6141.918899] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6141.922804] LustreError: 292890:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 6141.925823] LustreError: 292890:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6141.928422] LustreError: 292890:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6141.955075] Lustre: server umount lustre-OST0000 complete [ 6141.957180] LustreError: 292890:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6144.479985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6144.483192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.487680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6144.490747] Lustre: Skipped 1 previous similar message [ 6150.488255] Lustre: server umount lustre-MDT0000 complete [ 6151.880236] LustreError: 293258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892665 with bad export cookie 1664213907435170483 [ 6151.883819] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6151.884713] LustreError: 293258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6151.998918] Lustre: server umount lustre-MDT0001 complete [ 6155.904181] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6157.127686] Key type lgssc unregistered [ 6157.256451] LNet: 293992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6157.259765] LNetError: 293992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6157.273693] LNet: Removed LNI 192.168.202.152@tcp [ 6157.614682] Key type .llcrypt unregistered [ 6157.616136] Key type ._llcrypt unregistered [ 6161.642884] Key type ._llcrypt registered [ 6161.643751] Key type .llcrypt registered [ 6161.690897] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6161.998661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6162.005385] alg: No test for adler32 (adler32-zlib) [ 6162.854247] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6162.943626] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6164.527136] Key type lgssc registered [ 6164.901724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6200.501282] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6203.746043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6203.755851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.853194] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6204.863724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6204.929780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.274337] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6209.350275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.414885] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6209.427191] Lustre: 296705:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6209.538525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.973928] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6212.829565] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6214.154092] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6227.516232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6227.586252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6227.651246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.512800] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6231.362546] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6232.092795] Lustre: server umount lustre-OST0000 complete [ 6235.103866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6235.106669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.114117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.177231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.177774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.444559] Lustre: server umount lustre-MDT0000 complete [ 6240.685112] LustreError: 298343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892753 with bad export cookie 5975754603263535774 [ 6240.688137] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.689043] LustreError: 298343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6240.791510] Lustre: server umount lustre-MDT0001 complete [ 6244.162926] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6245.104727] Key type lgssc unregistered [ 6245.232539] LNet: 299916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6245.234728] LNetError: 299916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6245.245316] LNet: Removed LNI 192.168.202.152@tcp [ 6245.522878] Key type .llcrypt unregistered [ 6245.523657] Key type ._llcrypt unregistered [ 6254.388430] Key type ._llcrypt registered [ 6254.389443] Key type .llcrypt registered [ 6254.426219] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6254.849919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6254.857643] alg: No test for adler32 (adler32-zlib) [ 6255.718165] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6255.801853] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6257.383146] Key type lgssc registered [ 6257.736913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6260.236364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6260.242735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.397946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.581202] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6265.344330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.482358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6266.829247] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6268.594362] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6269.744150] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6271.916833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.970297] LustreError: 302885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.975597] LustreError: 302885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6272.027469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.961777] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6275.723185] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6280.161822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.163054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6280.166655] Lustre: Skipped 2 previous similar messages [ 6280.168760] Lustre: Skipped 1 previous similar message [ 6282.956052] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 6282.957984] Lustre: Skipped 1 previous similar message [ 6285.280884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.287209] Lustre: Skipped 2 previous similar messages [ 6285.920703] Lustre: server umount lustre-MDT0000 complete [ 6287.180456] LustreError: 301272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892800 with bad export cookie 13335258826778765253 [ 6287.183736] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6287.185847] LustreError: 301272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6288.067142] LustreError: 303535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.073311] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 6290.399992] LustreError: 303535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.400077] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.447768] LustreError: 303535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.447989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6292.457737] Lustre: Skipped 1 previous similar message [ 6293.353150] Lustre: server umount lustre-MDT0001 complete [ 6295.519378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6298.371324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.903805] LustreError: 300602:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d05362bd x1861109960168832/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6311.913639] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.918443] Lustre: MGS: Client b15aaed4-3b42-4421-b67b-6c61ccdbe73e (at 0@lo) reconnecting [ 6311.921697] Lustre: MGC192.168.202.152@tcp: Connection restored to 0@lo (at 0@lo) [ 6312.021282] LustreError: 302894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.027400] LustreError: 302894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6312.050435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.075780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6313.302528] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6316.238891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.391684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6317.413380] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6317.797388] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6319.632784] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6322.532591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6324.264572] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6325.474171] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6326.003285] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6327.777844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.778859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.779121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.779126] Lustre: Skipped 1 previous similar message [ 6327.782119] Lustre: Skipped 1 previous similar message [ 6333.335597] Lustre: server umount lustre-OST0000 complete [ 6336.991807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6336.994124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.764438] Lustre: server umount lustre-MDT0000 complete [ 6341.995260] LustreError: 301271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892855 with bad export cookie 13335258826778772407 [ 6341.998929] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6342.000974] LustreError: 301271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6342.121559] Lustre: server umount lustre-MDT0001 complete [ 6345.637069] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6346.682386] Key type lgssc unregistered [ 6346.801330] LNet: 306977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6346.803489] LNetError: 306977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6346.809351] LNet: Removed LNI 192.168.202.152@tcp [ 6347.093332] Key type .llcrypt unregistered [ 6347.094527] Key type ._llcrypt unregistered [ 6355.858603] Key type ._llcrypt registered [ 6355.859715] Key type .llcrypt registered [ 6355.893974] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6356.228601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6356.288623] alg: No test for adler32 (adler32-zlib) [ 6357.155473] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6357.235512] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6358.815205] Key type lgssc registered [ 6359.188368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6361.810701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6361.819015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.971549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.234711] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6367.000091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.127800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6368.431171] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6370.221971] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6371.410466] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6377.440662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6377.440853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.441536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6377.449129] Lustre: Skipped 1 previous similar message [ 6378.464315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.466182] Lustre: Skipped 2 previous similar messages [ 6378.518651] Lustre: server umount lustre-MDT0000 complete [ 6379.723494] LustreError: 308332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892892 with bad export cookie 2247523269150188008 [ 6379.726833] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6379.728485] LustreError: 308332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6379.842497] Lustre: server umount lustre-MDT0001 complete [ 6384.752289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.862058] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6384.873428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6384.944284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.242332] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6389.316497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.355454] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6389.368316] Lustre: 310712:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6389.474520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6390.820886] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6392.801766] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6394.059149] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6396.755366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6396.835692] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6396.840008] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6396.844698] LustreError: 312366:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 6396.847473] LustreError: 312366:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6396.849650] LustreError: 312366:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6396.878732] Lustre: server umount lustre-OST0000 complete [ 6396.880243] LustreError: 312366:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6399.679832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.745674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6399.807624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.827324] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6403.641747] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6406.229554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.260759] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6406.287547] Lustre: lustre-OST0001: new disk, initializing [ 6406.290287] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6406.376322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6406.382738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6406.400508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6406.440247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6408.315113] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6410.169032] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6411.745492] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6411.746497] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6411.747845] LustreError: Skipped 1 previous similar message [ 6411.752343] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6411.755767] Lustre: Skipped 1 previous similar message [ 6413.183892] Lustre: server umount lustre-OST0001 complete [ 6416.866823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6416.866861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.867055] LustreError: 312832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.867061] LustreError: 312832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6416.868144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.869257] LustreError: Skipped 1 previous similar message [ 6416.873885] Lustre: Skipped 2 previous similar messages [ 6420.498713] Lustre: server umount lustre-OST0000 complete [ 6421.984537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.985116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6421.988847] Lustre: Skipped 1 previous similar message [ 6421.990884] Lustre: Skipped 1 previous similar message [ 6427.872842] Lustre: server umount lustre-MDT0000 complete [ 6429.154990] LustreError: 310703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892942 with bad export cookie 2247523269150188547 [ 6429.158790] LustreError: 310703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6429.158807] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6429.273584] Lustre: server umount lustre-MDT0001 complete [ 6432.885184] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6433.897294] Key type lgssc unregistered [ 6434.024350] LNet: 315610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6434.027264] LNetError: 315610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6434.037356] LNet: Removed LNI 192.168.202.152@tcp [ 6434.327471] Key type .llcrypt unregistered [ 6434.328706] Key type ._llcrypt unregistered [ 6464.900201] Key type ._llcrypt registered [ 6464.901578] Key type .llcrypt registered [ 6464.946463] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6471.228535] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6471.559679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6471.573054] alg: No test for adler32 (adler32-zlib) [ 6472.420536] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6472.502049] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6474.079153] Key type lgssc registered [ 6474.435889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6485.342215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6485.347151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.437628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6486.451337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6486.487661] Lustre: lustre-MDT0000: new disk, initializing [ 6486.512201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.520156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6487.801143] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6492.035663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6492.076427] Lustre: 320497:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6492.099145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6492.100938] Lustre: Skipped 1 previous similar message [ 6492.134821] Lustre: lustre-MDT0001: new disk, initializing [ 6492.169357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.180386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6492.183186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6493.467803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6496.482573] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6497.630496] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6499.895296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6499.998451] Lustre: lustre-OST0000: new disk, initializing [ 6500.001050] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6500.029683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.664679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6500.668675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6500.733638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6501.983227] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6505.067708] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.394447] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6506.468394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6507.807216] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6507.879609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6511.074250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6511.074333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.076624] LustreError: Skipped 1 previous similar message [ 6511.081656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6511.084062] Lustre: Skipped 1 previous similar message [ 6514.637586] Lustre: server umount lustre-OST0000 complete [ 6516.192759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.193396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.196821] Lustre: Skipped 2 previous similar messages [ 6521.311952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6521.313817] Lustre: Skipped 2 previous similar messages [ 6521.941428] Lustre: server umount lustre-MDT0000 complete [ 6523.130121] LustreError: 320532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893036 with bad export cookie 11836809235618347048 [ 6523.132991] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.135878] LustreError: 320532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6523.242266] Lustre: server umount lustre-MDT0001 complete [ 6525.865663] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:50:37 (1774893037) [ 6528.274495] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:50:40 (1774893040) [ 6531.100203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.276366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6532.540635] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6535.319769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.444418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6536.727937] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6538.494480] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6539.631411] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6541.816229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.705726] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6545.401220] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6564.334877] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6564.859039] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6565.378545] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6565.898755] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6566.543403] Lustre: Failing over lustre-MDT0000 [ 6566.684907] Lustre: server umount lustre-MDT0000 complete [ 6567.904729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6567.906055] LustreError: 324833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.908021] Lustre: Skipped 2 previous similar messages [ 6567.912882] LustreError: 324833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6571.209850] LustreError: 325551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.024343] LustreError: 324839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.029627] LustreError: 324839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6576.323587] LustreError: 325551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.463228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.503730] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6579.606710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6579.608598] Lustre: Skipped 1 previous similar message [ 6579.639456] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6580.908576] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6581.444244] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6584.804794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6584.813149] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6584.828687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6585.935414] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6586.434334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6588.223115] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6588.865945] Lustre: Failing over lustre-MDT0000 [ 6589.025824] Lustre: server umount lustre-MDT0000 complete [ 6589.922522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.924083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6589.924829] LustreError: 324838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.924837] LustreError: 324838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6589.928815] Lustre: Skipped 2 previous similar messages [ 6600.159751] LustreError: 324838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.164763] LustreError: 324838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6601.700908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.736050] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6601.814438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.837026] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6601.923333] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6603.022756] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6606.820500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6606.822990] Lustre: Skipped 2 previous similar messages [ 6606.830344] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6606.844357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6607.871105] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6608.395216] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6611.938940] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.938972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6611.939807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.943460] Lustre: Skipped 1 previous similar message [ 6616.723129] Lustre: server umount lustre-MDT0000 complete [ 6617.056101] LustreError: 324838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.062445] LustreError: 324838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6617.957317] LustreError: 324819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893131 with bad export cookie 11836809235618349729 [ 6617.961605] LustreError: 324819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6617.962727] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.081351] Lustre: server umount lustre-MDT0001 complete [ 6629.596269] Lustre: server umount lustre-OST0000 complete [ 6634.568153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.745951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.991569] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6638.836024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.126200] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6641.709984] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6642.698106] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6644.772248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.476649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6648.005975] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6659.237555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6669.794263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.795480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.796601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.796608] Lustre: Skipped 2 previous similar messages [ 6669.798476] LustreError: Skipped 1 previous similar message [ 6669.803053] Lustre: Skipped 1 previous similar message [ 6671.114141] Lustre: server umount lustre-OST0000 complete [ 6674.912511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6678.420912] Lustre: server umount lustre-MDT0000 complete [ 6679.574879] LustreError: 330615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893192 with bad export cookie 11836809235618350478 [ 6679.578336] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.579512] LustreError: 330615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6679.682330] Lustre: server umount lustre-MDT0001 complete [ 6682.982441] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6683.868746] Key type lgssc unregistered [ 6683.989332] LNet: 333837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6683.991506] LNetError: 333837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6683.999340] LNet: Removed LNI 192.168.202.152@tcp [ 6684.265520] Key type .llcrypt unregistered [ 6684.266855] Key type ._llcrypt unregistered [ 6693.286511] Key type ._llcrypt registered [ 6693.287655] Key type .llcrypt registered [ 6693.324721] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6693.727757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6693.739529] alg: No test for adler32 (adler32-zlib) [ 6694.586711] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6694.667071] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6696.247150] Key type lgssc registered [ 6696.569204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6698.889731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6698.897036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.050817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.218205] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6703.914142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.041547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6705.432170] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6707.263951] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6708.480302] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6710.769925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.881523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.830143] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6714.687821] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6725.222969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6729.696222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6729.698547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.702964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.721043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.721759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.723032] Lustre: Skipped 1 previous similar message [ 6730.726429] Lustre: Skipped 1 previous similar message [ 6735.255387] Lustre: server umount lustre-MDT0000 complete [ 6735.840531] LustreError: 337488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.844592] LustreError: 337488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6736.446272] LustreError: 337493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893249 with bad export cookie 2133771435379176185 [ 6736.451176] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6736.557710] Lustre: server umount lustre-MDT0001 complete [ 6748.061189] Lustre: server umount lustre-OST0000 complete [ 6752.509500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.673194] LustreError: 338724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.700770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.114935] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6757.041612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.176657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6758.539312] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6760.462941] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6761.701986] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6763.995040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.040960] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6767.773867] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6777.446160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6788.065696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.066146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6788.066778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6788.066782] Lustre: Skipped 1 previous similar message [ 6788.069758] Lustre: Skipped 1 previous similar message [ 6790.803421] Lustre: server umount lustre-MDT0000 complete [ 6792.001084] LustreError: 338704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893305 with bad export cookie 2133771435379177284 [ 6792.002930] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6792.005268] LustreError: 338704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6792.104395] Lustre: server umount lustre-MDT0001 complete [ 6803.473613] Lustre: server umount lustre-OST0000 complete [ 6808.704169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6808.864581] LustreError: 342316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.888738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6808.891409] Lustre: Skipped 1 previous similar message [ 6810.201460] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6813.150353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.614261] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6816.436569] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6817.640394] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6819.957108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6821.973308] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6823.748282] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6835.430179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6840.802130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.802384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6840.806147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6840.808946] Lustre: Skipped 3 previous similar messages [ 6845.921690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6845.924390] Lustre: Skipped 1 previous similar message [ 6846.800156] Lustre: server umount lustre-OST0000 complete [ 6848.991775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6848.993864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6848.996889] Lustre: Skipped 1 previous similar message [ 6856.159971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.162072] Lustre: Skipped 4 previous similar messages [ 6862.303089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6862.358431] Lustre: server umount lustre-MDT0000 complete [ 6863.588093] LustreError: 343932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893376 with bad export cookie 2133771435379178292 [ 6863.590992] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6863.592333] LustreError: 343932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6863.716628] Lustre: server umount lustre-MDT0001 complete [ 6867.102565] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6868.045061] Key type lgssc unregistered [ 6868.167340] LNet: 345516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6868.170167] LNetError: 345516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6868.177631] LNet: Removed LNI 192.168.202.152@tcp [ 6868.445755] Key type .llcrypt unregistered [ 6868.446587] Key type ._llcrypt unregistered [ 6876.959044] Key type ._llcrypt registered [ 6876.960409] Key type .llcrypt registered [ 6876.995037] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6877.283966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6877.317032] alg: No test for adler32 (adler32-zlib) [ 6878.161700] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6878.229076] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6879.799138] Key type lgssc registered [ 6880.190331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6882.716536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6882.722923] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.725974] LustreError: 346916:0:(osd_handler.c:8556:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6882.734055] LustreError: 346916:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6882.736602] LustreError: 346916:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6882.739395] LustreError: 346916:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6882.742501] LustreError: 346916:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6885.252362] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6885.256056] LustreError: 347312:0:(osd_handler.c:8556:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6885.266196] LustreError: 347312:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6885.269497] LustreError: 347312:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6885.272893] LustreError: 347312:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6885.276781] LustreError: 347312:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6891.547943] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6892.501364] Key type lgssc unregistered [ 6892.614383] LNet: 348296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6892.616666] LNetError: 348296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6892.627301] LNet: Removed LNI 192.168.202.152@tcp [ 6892.888326] Key type .llcrypt unregistered [ 6892.889300] Key type ._llcrypt unregistered [ 6898.364095] Key type ._llcrypt registered [ 6898.365099] Key type .llcrypt registered [ 6898.397610] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6904.058104] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6904.404263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6904.430680] alg: No test for adler32 (adler32-zlib) [ 6905.272133] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6905.352544] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6906.927132] Key type lgssc registered [ 6907.282412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6917.518138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6917.522713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.605418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6918.615430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6918.647291] Lustre: lustre-MDT0000: new disk, initializing [ 6918.668843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.675819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6919.835833] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6923.715850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.752841] Lustre: 351391:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6923.773122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6923.774668] Lustre: Skipped 1 previous similar message [ 6923.806132] Lustre: lustre-MDT0001: new disk, initializing [ 6923.837268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6923.850562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6923.853484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6925.060726] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6927.976237] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6929.081405] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6931.102462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6931.196381] Lustre: lustre-OST0000: new disk, initializing [ 6931.198818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6931.226993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.930200] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6934.710320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6934.713875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6934.743909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6935.902420] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6937.177282] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6937.241176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6938.482292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6938.552017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6940.130016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.130300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6940.135809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6940.137373] LustreError: Skipped 1 previous similar message [ 6941.262971] Lustre: server umount lustre-OST0000 complete [ 6944.223828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6944.225904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6944.229522] Lustre: Skipped 1 previous similar message [ 6944.231025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6944.232583] Lustre: Skipped 1 previous similar message [ 6945.248078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6945.248639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.567144] Lustre: server umount lustre-MDT0000 complete [ 6949.727076] LustreError: 351384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893462 with bad export cookie 9865581358189685054 [ 6949.730423] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6949.730862] LustreError: 351384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6949.839530] Lustre: server umount lustre-MDT0001 complete [ 6952.435440] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6952.937294] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:57:45 (1774893465) [ 6955.015852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.199800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.412254] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6959.123395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6959.246775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.509291] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6962.137126] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6963.196217] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6965.251617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6967.003212] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6968.602993] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6970.669458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.733779] Lustre: lustre-OST0001: new disk, initializing [ 6970.736873] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6970.742343] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 6970.744601] Lustre: Skipped 1 previous similar message [ 6970.780868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6970.783331] Lustre: Skipped 1 previous similar message [ 6972.443444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6975.254143] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6975.975793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6975.978104] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6975.988180] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6981.088065] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6981.088285] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.092025] LustreError: Skipped 1 previous similar message [ 6981.096699] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6981.099636] Lustre: Skipped 1 previous similar message [ 6983.101701] Lustre: server umount lustre-OST0001 complete [ 6986.211167] LustreError: 357030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.215059] LustreError: 357030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6991.331782] LustreError: 357031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.335577] LustreError: 357031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6995.425136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6995.428554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.433899] Lustre: Skipped 1 previous similar message [ 6995.436443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6995.439144] Lustre: Skipped 1 previous similar message [ 6996.450873] LustreError: 357030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.457704] LustreError: 357030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7001.570978] LustreError: 357030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.574467] LustreError: 357030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7006.690901] LustreError: 358858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.695773] LustreError: 358858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7009.247171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7009.300553] Lustre: server umount lustre-OST0000 complete [ 7010.784032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7010.788156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7010.793630] Lustre: Skipped 1 previous similar message [ 7011.808487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.811109] Lustre: Skipped 8 previous similar messages [ 7016.723049] Lustre: server umount lustre-MDT0000 complete [ 7016.928068] LustreError: 355437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.935498] LustreError: 355437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7018.019318] LustreError: 357929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893531 with bad export cookie 9865581358189685922 [ 7018.023036] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7018.024853] LustreError: 357929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7032.287086] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7032.373056] Lustre: server umount lustre-MDT0001 complete [ 7035.409934] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7036.329305] Key type lgssc unregistered [ 7036.446402] LNet: 359800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7036.448659] LNetError: 359800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7036.454253] LNet: Removed LNI 192.168.202.152@tcp [ 7036.736635] Key type .llcrypt unregistered [ 7036.737982] Key type ._llcrypt unregistered [ 7052.755083] Key type ._llcrypt registered [ 7052.756198] Key type .llcrypt registered [ 7052.792752] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7053.162186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7053.203422] alg: No test for adler32 (adler32-zlib) [ 7054.045728] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7054.117942] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7055.695106] Key type lgssc registered [ 7056.024602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7058.267830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7058.273109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.358091] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7059.367892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7059.427782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.723273] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7063.591918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.624171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7063.636551] Lustre: 362555:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7063.742345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7065.060163] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7066.860311] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7068.058988] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7069.151989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7069.154439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.158601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.176966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.177487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.903655] Lustre: server umount lustre-MDT0000 complete [ 7076.170425] LustreError: 362547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893589 with bad export cookie 6772052966188443472 [ 7076.174424] LustreError: 362547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7076.174498] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.292255] Lustre: server umount lustre-MDT0001 complete [ 7079.431870] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:59:51 (1774893591) [ 7081.700030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.871352] LustreError: 364838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.876704] LustreError: 364838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7081.894734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.181616] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7085.814890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.925434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7087.153288] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7088.808541] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7089.894875] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7091.944262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7092.007193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7092.066832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.776976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7095.380737] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7098.892966] Lustre: Permanently deactivating lustre-OST0000 [ 7098.898034] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7103.981833] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7104.500218] LustreError: 367481:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7104.502834] LustreError: 367481:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7104.504421] LustreError: 367481:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7105.026882] LustreError: 367530:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7105.028892] LustreError: 367530:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7111.701929] Lustre: server umount lustre-OST0000 complete [ 7112.591539] LustreError: 367780:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7112.595026] LustreError: 367780:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7115.232392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.233456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.236072] Lustre: Skipped 1 previous similar message [ 7115.238505] Lustre: Skipped 1 previous similar message [ 7120.351902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7120.353958] Lustre: Skipped 2 previous similar messages [ 7123.447943] Lustre: server umount lustre-MDT0000 complete [ 7124.646186] LustreError: 364817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893637 with bad export cookie 6772052966188444039 [ 7124.649426] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7124.650038] LustreError: 364817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7134.223514] Lustre: server umount lustre-MDT0001 complete [ 7137.069916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.328314] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7139.213611] LustreError: 368920:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7141.397223] Lustre: 369258:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.152@tcp [ 7142.122142] Lustre: server umount MGS complete [ 7146.099699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.259464] LustreError: 369981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.263855] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7146.266951] Lustre: Skipped 1 previous similar message [ 7146.285233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.487109] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7150.091038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.176331] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7151.359951] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7152.997511] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7154.116786] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7154.737723] Lustre: Permanently reactivating lustre-OST0000 [ 7154.742623] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7154.745526] Lustre: Skipped 2 previous similar messages [ 7175.135968] LustreError: 371395:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7177.385521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.509755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.511519] Lustre: Skipped 1 previous similar message [ 7179.209266] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7180.880829] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7195.106235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.106455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.107380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7195.114047] Lustre: Skipped 1 previous similar message [ 7198.471285] Lustre: server umount lustre-OST0000 complete [ 7200.224509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.344089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.346210] Lustre: Skipped 4 previous similar messages [ 7205.712849] Lustre: server umount lustre-MDT0000 complete [ 7206.923891] LustreError: 373380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893720 with bad export cookie 6772052966188445075 [ 7206.926424] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.926917] LustreError: 373380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7207.028125] Lustre: server umount lustre-MDT0001 complete [ 7210.293106] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7211.268750] Key type lgssc unregistered [ 7211.396463] LNet: 374113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7211.399968] LNetError: 374113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7211.408287] LNet: Removed LNI 192.168.202.152@tcp [ 7211.699225] Key type .llcrypt unregistered [ 7211.700442] Key type ._llcrypt unregistered [ 7217.093386] Key type ._llcrypt registered [ 7217.094263] Key type .llcrypt registered [ 7217.125623] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7222.639242] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7222.970298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7223.005294] alg: No test for adler32 (adler32-zlib) [ 7223.837250] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7223.906435] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7225.487135] Key type lgssc registered [ 7225.804122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7236.101581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7236.106487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.185736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7237.194601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7237.225092] Lustre: lustre-MDT0000: new disk, initializing [ 7237.244458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.250366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7238.421935] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7242.463267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.507566] Lustre: 377209:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7242.539779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7242.541966] Lustre: Skipped 1 previous similar message [ 7242.577090] Lustre: lustre-MDT0001: new disk, initializing [ 7242.608802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7242.620583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7242.625085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7243.840454] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7246.902988] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7248.132641] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7250.370327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.488197] Lustre: lustre-OST0000: new disk, initializing [ 7250.490084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7250.519278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7252.372121] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7252.536123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7252.540134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7252.571990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7255.465681] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7256.849274] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7256.922161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7258.259041] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7258.328485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7262.690296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.690849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7262.698783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7262.701916] LustreError: Skipped 1 previous similar message [ 7265.108620] Lustre: server umount lustre-OST0000 complete [ 7267.809427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.810670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.814637] Lustre: Skipped 2 previous similar messages [ 7267.818554] Lustre: Skipped 2 previous similar messages [ 7272.594528] Lustre: server umount lustre-MDT0000 complete [ 7273.728168] LustreError: 378951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893786 with bad export cookie 15198610765351207640 [ 7273.730800] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.731615] LustreError: 378951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7273.838287] Lustre: server umount lustre-MDT0001 complete [ 7276.348510] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7276.868960] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:03:09 (1774893789) [ 7278.878930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.005531] LustreError: 381260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.010744] LustreError: 381260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7279.026203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7280.246636] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7282.867037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7284.183367] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7285.750324] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7286.919217] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7289.126872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7289.233491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.235419] Lustre: Skipped 1 previous similar message [ 7291.109251] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7292.882400] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7293.527580] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7294.977789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7298.530672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.532514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7298.534648] Lustre: Skipped 1 previous similar message [ 7298.534890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7298.536320] Lustre: Skipped 1 previous similar message [ 7302.286502] Lustre: server umount lustre-OST0000 complete [ 7303.648532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7303.648737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.649513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.657258] Lustre: Skipped 1 previous similar message [ 7308.768336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7308.771097] Lustre: Skipped 2 previous similar messages [ 7309.652725] Lustre: server umount lustre-MDT0000 complete [ 7310.864721] LustreError: 381241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893824 with bad export cookie 15198610765351208508 [ 7310.868670] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.870039] LustreError: 381241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7310.989443] Lustre: server umount lustre-MDT0001 complete [ 7314.474289] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7315.414128] Key type lgssc unregistered [ 7315.548300] LNet: 384627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7315.550251] LNetError: 384627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7315.558281] LNet: Removed LNI 192.168.202.152@tcp [ 7315.820813] Key type .llcrypt unregistered [ 7315.821802] Key type ._llcrypt unregistered [ 7322.854147] Key type ._llcrypt registered [ 7322.854936] Key type .llcrypt registered [ 7322.889644] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7322.995556] Key type .llcrypt unregistered [ 7322.997185] Key type ._llcrypt unregistered [ 7329.922886] Key type ._llcrypt registered [ 7329.923829] Key type .llcrypt registered [ 7329.957254] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7330.376111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7330.386478] alg: No test for adler32 (adler32-zlib) [ 7331.232186] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7331.314706] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7332.903172] Key type lgssc registered [ 7333.261238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7335.840372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7335.846191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.006994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7338.271901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7341.069273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.184687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.025646] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7344.817255] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7346.656559] LustreError: 386780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.661787] LustreError: 386780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7347.018918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.150598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.200505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7348.416525] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7352.289754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7352.293519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7352.299590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7353.313827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7353.315160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7353.315620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.392402] Lustre: server umount lustre-OST0000 complete [ 7362.017600] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7362.019966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.023911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.025389] Lustre: Skipped 1 previous similar message [ 7364.756370] Lustre: server umount lustre-MDT0000 complete [ 7365.968135] LustreError: 388395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893879 with bad export cookie 483346193098838222 [ 7365.971813] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.973743] LustreError: 388395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7366.081745] Lustre: server umount lustre-MDT0001 complete [ 7369.657901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7370.686798] Key type lgssc unregistered [ 7370.803357] LNet: 389766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7370.806126] LNetError: 389766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7370.815291] LNet: Removed LNI 192.168.202.152@tcp [ 7371.131509] Key type .llcrypt unregistered [ 7371.133519] Key type ._llcrypt unregistered [ 7380.011918] Key type ._llcrypt registered [ 7380.012966] Key type .llcrypt registered [ 7380.051991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7380.536893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7380.544843] alg: No test for adler32 (adler32-zlib) [ 7381.414131] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7381.508327] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7383.095106] Key type lgssc registered [ 7383.441067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7386.008146] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7386.014142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.439368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7406.559177] LustreError: 391116:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 7433.183178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7435.135400] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7437.007148] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7439.261883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.430219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.450452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7440.723665] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7443.530319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.665095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7445.000585] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7446.698265] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7447.801456] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7454.179295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7454.179390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.183410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7454.185767] Lustre: Skipped 1 previous similar message [ 7454.188288] Lustre: Skipped 1 previous similar message [ 7456.137560] Lustre: server umount lustre-OST0000 complete [ 7459.296883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7459.297278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.304082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.306330] Lustre: Skipped 1 previous similar message [ 7463.476979] Lustre: server umount lustre-MDT0000 complete [ 7464.417061] LustreError: 393601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.422641] LustreError: 393601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7464.679255] LustreError: 391122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893977 with bad export cookie 6941931371613307835 [ 7464.684590] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7464.803692] Lustre: server umount lustre-MDT0001 complete [ 7468.547190] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7469.592463] Key type lgssc unregistered [ 7469.726587] LNet: 394334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7469.730818] LNetError: 394334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.737420] LNet: Removed LNI 192.168.202.152@tcp [ 7470.051256] Key type .llcrypt unregistered [ 7470.053238] Key type ._llcrypt unregistered [ 7479.151656] Key type ._llcrypt registered [ 7479.153308] Key type .llcrypt registered [ 7479.198491] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7479.574145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7479.623422] alg: No test for adler32 (adler32-zlib) [ 7480.478657] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7480.571993] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7482.159157] Key type lgssc registered [ 7482.524908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7485.128513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7485.135861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.296945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.616778] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7490.435167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.558094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7491.865226] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7493.710952] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7494.901871] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7497.176993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.311784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.222141] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7501.037886] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7511.651429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7516.127820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7516.129654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.132808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.874787] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 7517.153499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.157691] Lustre: Skipped 1 previous similar message [ 7518.802301] Lustre: server umount lustre-MDT0000 complete [ 7519.504678] LustreError: 395689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.52@tcp arrived at 1774894032 with bad export cookie 195775428934239803 [ 7520.095116] LustreError: 395690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774894033 with bad export cookie 195775428934239180 [ 7520.098198] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.101231] LustreError: 395690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7521.247845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7521.251165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.256555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.258632] Lustre: Skipped 3 previous similar messages [ 7522.271817] LustreError: 396431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.276803] LustreError: 396431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7525.279809] LustreError: 396431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.280128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.285678] LustreError: 396431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7525.288015] Lustre: Skipped 1 previous similar message [ 7526.223197] Lustre: server umount lustre-OST0000 complete [ 7528.306485] Lustre: server umount lustre-MDT0001 complete [ 7531.432368] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7532.347694] Key type lgssc unregistered [ 7532.478392] LNet: 399060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7532.480210] LNetError: 399060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7532.486259] LNet: Removed LNI 192.168.202.152@tcp [ 7532.778328] Key type .llcrypt unregistered [ 7532.779318] Key type ._llcrypt unregistered [ 7541.297343] Key type ._llcrypt registered [ 7541.298299] Key type .llcrypt registered [ 7541.333975] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7541.781509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.788273] alg: No test for adler32 (adler32-zlib) [ 7542.643320] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7542.727811] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7544.311136] Key type lgssc registered [ 7544.696565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7547.324339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7547.332951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.505598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.752365] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7552.517994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.639235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7553.927911] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7555.561492] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7556.688604] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7558.890524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.009370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.891479] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7562.654236] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7573.350547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7583.434223] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 7583.718159] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7583.718291] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.724315] Lustre: Skipped 1 previous similar message [ 7585.297206] Lustre: server umount lustre-MDT0001 complete [ 7588.832311] LustreError: 400434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7588.833103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7588.833623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7588.834186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7588.834189] Lustre: Skipped 2 previous similar messages [ 7588.837275] LustreError: 400434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7593.114911] Lustre: server umount lustre-OST0000 complete [ 7593.952486] LustreError: 400434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.382708] Lustre: server umount lustre-MDT0000 complete [ 7598.393701] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7599.342017] Key type lgssc unregistered [ 7599.454420] LNet: 403783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7599.456430] LNetError: 403783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7599.468325] LNet: Removed LNI 192.168.202.152@tcp [ 7599.742417] Key type .llcrypt unregistered [ 7599.743270] Key type ._llcrypt unregistered [ 7607.066508] Key type ._llcrypt registered [ 7607.067358] Key type .llcrypt registered [ 7607.107646] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 7612.810428] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7613.215688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7613.282509] alg: No test for adler32 (adler32-zlib) [ 7614.156721] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7614.246243] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7615.831256] Key type lgssc registered [ 7616.190860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.115252] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7631.805091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7631.811864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.910677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7632.922739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7632.960985] Lustre: lustre-MDT0000: new disk, initializing [ 7632.985362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7632.991735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7634.337515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7640.008747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.116821] Lustre: lustre-OST0000: new disk, initializing [ 7640.119873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7640.123233] Lustre: Skipped 1 previous similar message [ 7640.150580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.598921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7640.602562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7640.610740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7642.068561] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7645.097192] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7650.384808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.431150] Lustre: 407718:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7650.457313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7650.489089] Lustre: lustre-MDT0001: new disk, initializing [ 7650.527346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7650.538979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7650.543046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7651.751598] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7655.241875] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7655.313698] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7656.517543] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7656.586439] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7658.698429] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 7661.536506] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7661.537063] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.543934] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7661.546371] Lustre: Skipped 1 previous similar message [ 7663.812492] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 7664.218927] Lustre: server umount lustre-MDT0001 complete [ 7671.778410] LustreError: 407726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.786027] LustreError: 407726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7676.896109] LustreError: 407729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.504058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7681.507793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.512852] Lustre: Skipped 1 previous similar message [ 7681.515511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.017540] LustreError: 407729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.582903] Lustre: server umount lustre-OST0000 complete [ 7698.399107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7698.477847] Lustre: server umount lustre-MDT0000 complete [ 7702.318793] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7703.257892] Key type lgssc unregistered [ 7703.373327] LNet: 412285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7703.375601] LNetError: 412285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7703.382304] LNet: Removed LNI 192.168.202.152@tcp [ 7703.650139] Key type .llcrypt unregistered [ 7703.651259] Key type ._llcrypt unregistered [ 7715.349608] Key type ._llcrypt registered [ 7715.350608] Key type .llcrypt registered [ 7715.388184] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7715.487252] Key type .llcrypt unregistered [ 7715.488567] Key type ._llcrypt unregistered [ 7721.472335] Key type ._llcrypt registered [ 7721.473560] Key type .llcrypt registered [ 7721.508529] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7721.887296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7721.897241] alg: No test for adler32 (adler32-zlib) [ 7722.743088] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7722.829407] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7724.407136] Key type lgssc registered [ 7724.762462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7732.900787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7732.906263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7734.006698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7734.018740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7734.062882] Lustre: lustre-MDT0000: new disk, initializing [ 7734.088738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7734.097192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7735.362317] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7739.289578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.331071] Lustre: 415600:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7739.367266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7739.369534] Lustre: Skipped 1 previous similar message [ 7739.408623] Lustre: lustre-MDT0001: new disk, initializing [ 7739.443488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7739.457873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7739.462117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7740.668872] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7743.621471] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7744.665427] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7746.768847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7746.874159] Lustre: lustre-OST0000: new disk, initializing [ 7746.875684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7746.905442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7748.630481] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7751.571768] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7755.254400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7755.257503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7755.290068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7759.840126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7759.842644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7759.847110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.354231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.354303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.359386] Lustre: Skipped 2 previous similar messages [ 7765.135087] Lustre: server umount lustre-MDT0000 complete [ 7765.472476] LustreError: 417780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.478313] LustreError: 417780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7766.344227] LustreError: 415591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774894279 with bad export cookie 14484085891012427458 [ 7766.346586] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7766.348567] LustreError: 415591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7766.463798] Lustre: server umount lustre-MDT0001 complete [ 7777.816539] Lustre: server umount lustre-OST0000 complete [ 7781.312794] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:11:33 (1774894293) [ 7783.963346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.144237] LustreError: 419259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.170817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7785.474793] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7788.421726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7788.560711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7790.001913] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7791.816397] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7793.051582] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7806.339730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7806.412095] Lustre: Found index 0 for lustre-OST0000, updating log [ 7806.467800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7808.229544] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7809.573919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7809.863839] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7813.026544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.027109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.033668] Lustre: Skipped 1 previous similar message [ 7815.151225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.153579] Lustre: Skipped 1 previous similar message [ 7817.107538] Lustre: server umount lustre-OST0000 complete [ 7829.681936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7829.688273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7829.691015] Lustre: Skipped 2 previous similar messages [ 7834.591843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7834.594839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.863962] Lustre: server umount lustre-MDT0000 complete [ 7837.148341] LustreError: 419241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774894350 with bad export cookie 14484085891012428851 [ 7837.150344] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7837.153607] LustreError: 419241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7837.279663] Lustre: server umount lustre-MDT0001 complete [ 7839.957197] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:12:32 (1774894352) [ 7871.327297] LNet: 1 local NIs in recovery (showing 1): 192.168.202.152@tcp [ 8419.234276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.383934] LustreError: 425325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8419.407168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8420.683564] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8423.472065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.907444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8426.705600] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8427.867069] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8441.166090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8441.273602] Lustre: Found index 0 for lustre-OST0000, updating log [ 8443.208554] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8445.002134] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8445.749824] Lustre: server umount lustre-OST0000 complete [ 8449.056847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.061067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.062590] Lustre: Skipped 4 previous similar messages [ 8450.080182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8450.082081] Lustre: Skipped 1 previous similar message [ 8455.135991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.138016] Lustre: Skipped 1 previous similar message [ 8460.256132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8460.258453] Lustre: Skipped 1 previous similar message [ 8461.279097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8461.338270] Lustre: server umount lustre-MDT0000 complete [ 8462.525337] LustreError: 425305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774894975 with bad export cookie 14484085891012429908 [ 8462.528812] LustreError: 425305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8462.529122] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.637919] Lustre: server umount lustre-MDT0001 complete [ 8472.415230] LNet: 1 local NIs in recovery (showing 1): 192.168.202.152@tcp [ 8479.672826] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8482.436425] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8493.692709] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:23:25 (1774895005) [ 8498.986265] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8499.943707] Key type lgssc unregistered [ 8504.159826] LNet: 431572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8504.162046] LNetError: 431572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8570.725312] LNet: Removed LNI 192.168.202.152@tcp [ 8571.019251] Key type .llcrypt unregistered [ 8571.020376] Key type ._llcrypt unregistered [ 8577.530442] Key type ._llcrypt registered [ 8577.531326] Key type .llcrypt registered [ 8577.567835] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8577.873053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8577.881092] alg: No test for adler32 (adler32-zlib) [ 8578.721794] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 8578.801285] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8580.375152] Key type lgssc registered [ 8580.769085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8582.303053] LNet: Added LNI 192.168.202.152@tcp100 [8/256/0/180] [ 8582.597408] LNet: Added LNI 192.168.202.152@tcp101 [8/256/0/180] [ 8582.911750] LNet: Added LNI 192.168.202.152@tcp102 [8/256/0/180] [ 8583.209258] LNet: Added LNI 192.168.202.152@tcp103 [8/256/0/180] [ 8583.480582] LNet: Added LNI 192.168.202.152@tcp104 [8/256/0/180] [ 8583.765840] LNet: Added LNI 192.168.202.152@tcp105 [8/256/0/180] [ 8584.056385] LNet: Added LNI 192.168.202.152@tcp106 [8/256/0/180] [ 8584.338369] LNet: Added LNI 192.168.202.152@tcp107 [8/256/0/180] [ 8584.626441] LNet: Added LNI 192.168.202.152@tcp108 [8/256/0/180] [ 8584.915495] LNet: Added LNI 192.168.202.152@tcp109 [8/256/0/180] [ 8585.190440] LNet: Added LNI 192.168.202.152@tcp110 [8/256/0/180] [ 8585.476721] LNet: Added LNI 192.168.202.152@tcp111 [8/256/0/180] [ 8585.765855] LNet: Added LNI 192.168.202.152@tcp112 [8/256/0/180] [ 8586.055895] LNet: Added LNI 192.168.202.152@tcp113 [8/256/0/180] [ 8586.338173] LNet: Added LNI 192.168.202.152@tcp114 [8/256/0/180] [ 8586.624184] LNet: Added LNI 192.168.202.152@tcp115 [8/256/0/180] [ 8586.911638] LNet: Added LNI 192.168.202.152@tcp116 [8/256/0/180] [ 8587.191864] LNet: Added LNI 192.168.202.152@tcp117 [8/256/0/180] [ 8587.482461] LNet: Added LNI 192.168.202.152@tcp118 [8/256/0/180] [ 8587.760578] LNet: Added LNI 192.168.202.152@tcp119 [8/256/0/180] [ 8588.045090] LNet: Added LNI 192.168.202.152@tcp120 [8/256/0/180] [ 8588.350680] LNet: Added LNI 192.168.202.152@tcp121 [8/256/0/180] [ 8588.644921] LNet: Added LNI 192.168.202.152@tcp122 [8/256/0/180] [ 8588.967550] LNet: Added LNI 192.168.202.152@tcp123 [8/256/0/180] [ 8589.268343] LNet: Added LNI 192.168.202.152@tcp124 [8/256/0/180] [ 8589.545698] LNet: Added LNI 192.168.202.152@tcp125 [8/256/0/180] [ 8589.828718] LNet: Added LNI 192.168.202.152@tcp126 [8/256/0/180] [ 8590.107080] LNet: Added LNI 192.168.202.152@tcp127 [8/256/0/180] [ 8590.384603] LNet: Added LNI 192.168.202.152@tcp128 [8/256/0/180] [ 8590.669483] LNet: Added LNI 192.168.202.152@tcp129 [8/256/0/180] [ 8590.949139] LNet: Added LNI 192.168.202.152@tcp130 [8/256/0/180] [ 8591.235924] LNet: Added LNI 192.168.202.152@tcp131 [8/256/0/180] [ 8591.531078] LNet: Added LNI 192.168.202.152@tcp132 [8/256/0/180] [ 8591.816811] LNet: Added LNI 192.168.202.152@tcp133 [8/256/0/180] [ 8592.108180] LNet: Added LNI 192.168.202.152@tcp134 [8/256/0/180] [ 8592.390094] LNet: Added LNI 192.168.202.152@tcp135 [8/256/0/180] [ 8594.484397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8594.489913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.537561] Lustre: 434550:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [ 8595.605804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8595.616564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8595.657549] Lustre: lustre-MDT0000: new disk, initializing [ 8595.694400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8595.701791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8597.021953] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8601.155877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.216926] Lustre: 434569:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.251986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8601.254592] Lustre: Skipped 1 previous similar message [ 8601.297965] Lustre: lustre-MDT0001: new disk, initializing [ 8601.311331] Lustre: 435351:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [ 8601.314852] Lustre: 435351:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8601.362841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8601.377621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8601.381479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8602.666784] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8602.929303] Lustre: 435628:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [ 8602.933064] Lustre: 435628:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8605.615691] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8606.799183] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8608.937627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8609.039368] Lustre: lustre-OST0000: new disk, initializing [ 8609.041155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8609.073854] Lustre: 436318:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [ 8609.077822] Lustre: 436318:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8609.099584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.859343] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8613.771748] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8615.923917] Lustre: server umount lustre-OST0000 complete [ 8616.948354] Lustre: 437323:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [ 8616.952020] Lustre: 437323:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8622.048762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.048915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8622.049484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8622.052187] Lustre: Skipped 1 previous similar message [ 8623.191893] Lustre: server umount lustre-MDT0000 complete [ 8624.351440] LustreError: 434559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774895137 with bad export cookie 14911865047424207834 [ 8624.354566] LustreError: 434559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8624.356490] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8624.473436] Lustre: server umount lustre-MDT0001 complete [ 8627.719831] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8628.761929] Key type lgssc unregistered [ 8628.928481] LNet: 438092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8628.931364] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8628.933405] LNet: Removed LNI 192.168.202.152@tcp [ 8629.983166] LNet: Removed LNI 192.168.202.152@tcp100 [ 8629.984418] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8629.986316] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8629.988350] LNet: Removed LNI 192.168.202.152@tcp101 [ 8631.007192] LNet: Removed LNI 192.168.202.152@tcp102 [ 8631.008544] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8631.010560] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8631.012540] LNet: Removed LNI 192.168.202.152@tcp103 [ 8632.031173] LNet: Removed LNI 192.168.202.152@tcp104 [ 8633.055186] LNet: Removed LNI 192.168.202.152@tcp105 [ 8633.056537] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8633.058850] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8633.060920] LNet: Removed LNI 192.168.202.152@tcp106 [ 8634.079135] LNet: Removed LNI 192.168.202.152@tcp107 [ 8635.103168] LNet: Removed LNI 192.168.202.152@tcp108 [ 8636.127112] LNet: Removed LNI 192.168.202.152@tcp109 [ 8637.151183] LNet: Removed LNI 192.168.202.152@tcp110 [ 8637.152571] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8637.154660] LNetError: 438092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8637.157017] LNet: Removed LNI 192.168.202.152@tcp111 [ 8638.175175] LNet: Removed LNI 192.168.202.152@tcp112 [ 8639.199185] LNet: Removed LNI 192.168.202.152@tcp113 [ 8640.223140] LNet: Removed LNI 192.168.202.152@tcp114 [ 8640.224361] LNet: Removed LNI 192.168.202.152@tcp115 [ 8640.225407] LNet: Removed LNI 192.168.202.152@tcp116 [ 8640.226403] LNet: Removed LNI 192.168.202.152@tcp117 [ 8640.227467] LNet: Removed LNI 192.168.202.152@tcp118 [ 8640.228504] LNet: Removed LNI 192.168.202.152@tcp119 [ 8640.229505] LNet: Removed LNI 192.168.202.152@tcp120 [ 8640.230542] LNet: Removed LNI 192.168.202.152@tcp121 [ 8640.231505] LNet: Removed LNI 192.168.202.152@tcp122 [ 8640.232879] LNet: Removed LNI 192.168.202.152@tcp123 [ 8640.234021] LNet: Removed LNI 192.168.202.152@tcp124 [ 8640.235080] LNet: Removed LNI 192.168.202.152@tcp125 [ 8640.236190] LNet: Removed LNI 192.168.202.152@tcp126 [ 8640.237190] LNet: Removed LNI 192.168.202.152@tcp127 [ 8640.238294] LNet: Removed LNI 192.168.202.152@tcp128 [ 8640.239720] LNet: Removed LNI 192.168.202.152@tcp129 [ 8640.240703] LNet: Removed LNI 192.168.202.152@tcp130 [ 8640.241717] LNet: Removed LNI 192.168.202.152@tcp131 [ 8640.242689] LNet: Removed LNI 192.168.202.152@tcp132 [ 8640.243721] LNet: Removed LNI 192.168.202.152@tcp133 [ 8640.244814] LNet: Removed LNI 192.168.202.152@tcp134 [ 8640.252511] LNet: Removed LNI 192.168.202.152@tcp135 [ 8640.505204] Key type .llcrypt unregistered [ 8640.506328] Key type ._llcrypt unregistered [ 8647.565944] Key type ._llcrypt registered [ 8647.567023] Key type .llcrypt registered [ 8647.610278] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8653.487226] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8653.959674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8653.975568] alg: No test for adler32 (adler32-zlib) [ 8654.821880] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 8654.897265] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8656.471080] Key type lgssc registered [ 8656.804574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8668.291785] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:26:20 (1774895180) [ 8674.434775] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8675.418915] Key type lgssc unregistered [ 8675.543366] LNet: 442096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8675.545354] LNetError: 442096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8675.553297] LNet: Removed LNI 192.168.202.152@tcp [ 8675.811468] Key type .llcrypt unregistered [ 8675.812440] Key type ._llcrypt unregistered [ 8681.905532] Key type ._llcrypt registered [ 8681.906487] Key type .llcrypt registered [ 8681.945661] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8682.368303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8682.374317] alg: No test for adler32 (adler32-zlib) [ 8683.228911] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 8683.304662] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8684.879128] Key type lgssc registered [ 8685.210689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8688.087766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8688.092661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.175818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8689.185722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8689.223296] Lustre: lustre-MDT0000: new disk, initializing [ 8689.245681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.251257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8690.471209] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8694.450430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8694.493181] Lustre: 443270:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8694.515163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8694.516647] Lustre: Skipped 1 previous similar message [ 8694.550048] Lustre: lustre-MDT0001: new disk, initializing [ 8694.583337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.595200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8694.597798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8695.771154] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8698.730431] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8699.790632] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8701.866045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8701.967118] Lustre: lustre-OST0000: new disk, initializing [ 8701.969235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8701.998363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8703.761819] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8706.037787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8706.041127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8706.077202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8706.726929] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8707.796107] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing lnet_if_list [ 8709.310490] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8709.322860] LNet: Added LNI 192.168.202.152@tcp10 [8/256/0/180] [ 8711.006351] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8711.016412] LNet: Added LNI 192.168.202.152@tcp11 [8/256/0/180] [ 8712.169109] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8712.178768] LNet: Added LNI 192.168.202.152@tcp12 [8/256/0/180] [ 8713.394230] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8713.404920] LNet: Added LNI 192.168.202.152@tcp13 [8/256/0/180] [ 8714.709654] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8714.723242] LNet: Added LNI 192.168.202.152@tcp14 [8/256/0/180] [ 8715.984044] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8715.995057] LNet: Added LNI 192.168.202.152@tcp15 [8/256/0/180] [ 8717.321113] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8717.330902] LNet: Added LNI 192.168.202.152@tcp16 [8/256/0/180] [ 8718.694026] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8718.708572] LNet: Added LNI 192.168.202.152@tcp17 [8/256/0/180] [ 8720.241951] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8720.250995] LNet: Added LNI 192.168.202.152@tcp18 [8/256/0/180] [ 8721.732350] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8721.741522] LNet: Added LNI 192.168.202.152@tcp19 [8/256/0/180] [ 8723.128332] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8723.137871] LNet: Added LNI 192.168.202.152@tcp20 [8/256/0/180] [ 8724.648813] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8724.661803] LNet: Added LNI 192.168.202.152@tcp21 [8/256/0/180] [ 8726.320169] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8726.330992] LNet: Added LNI 192.168.202.152@tcp22 [8/256/0/180] [ 8728.026830] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8728.040233] LNet: Added LNI 192.168.202.152@tcp23 [8/256/0/180] [ 8729.668391] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8729.678349] LNet: Added LNI 192.168.202.152@tcp24 [8/256/0/180] [ 8731.306261] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8731.316238] LNet: Added LNI 192.168.202.152@tcp25 [8/256/0/180] [ 8733.065156] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8733.075578] LNet: Added LNI 192.168.202.152@tcp26 [8/256/0/180] [ 8734.755868] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8734.764809] LNet: Added LNI 192.168.202.152@tcp27 [8/256/0/180] [ 8736.416248] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8736.428431] LNet: Added LNI 192.168.202.152@tcp28 [8/256/0/180] [ 8738.103901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8738.113311] LNet: Added LNI 192.168.202.152@tcp29 [8/256/0/180] [ 8739.836271] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8739.846616] LNet: Added LNI 192.168.202.152@tcp30 [8/256/0/180] [ 8741.613092] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8741.623274] LNet: Added LNI 192.168.202.152@tcp31 [8/256/0/180] [ 8743.453050] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8743.464699] LNet: Added LNI 192.168.202.152@tcp32 [8/256/0/180] [ 8745.331678] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8745.342378] LNet: Added LNI 192.168.202.152@tcp33 [8/256/0/180] [ 8747.222136] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8747.231506] LNet: Added LNI 192.168.202.152@tcp34 [8/256/0/180] [ 8749.108892] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8749.118814] LNet: Added LNI 192.168.202.152@tcp35 [8/256/0/180] [ 8751.062975] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8751.072901] LNet: Added LNI 192.168.202.152@tcp36 [8/256/0/180] [ 8753.165264] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8753.175454] LNet: Added LNI 192.168.202.152@tcp37 [8/256/0/180] [ 8755.194509] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8755.203767] LNet: Added LNI 192.168.202.152@tcp38 [8/256/0/180] [ 8757.211710] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8757.223761] LNet: Added LNI 192.168.202.152@tcp39 [8/256/0/180] [ 8759.250368] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8759.260131] LNet: Added LNI 192.168.202.152@tcp40 [8/256/0/180] [ 8761.347088] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8761.356325] LNet: Added LNI 192.168.202.152@tcp41 [8/256/0/180] [ 8763.651214] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8763.662146] LNet: Added LNI 192.168.202.152@tcp42 [8/256/0/180] [ 8765.975314] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8765.984474] LNet: Added LNI 192.168.202.152@tcp43 [8/256/0/180] [ 8768.390877] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8768.401844] LNet: Added LNI 192.168.202.152@tcp44 [8/256/0/180] [ 8770.783570] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8770.792859] LNet: Added LNI 192.168.202.152@tcp45 [8/256/0/180] [ 8773.165606] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8773.176829] LNet: Added LNI 192.168.202.152@tcp46 [8/256/0/180] [ 8775.856591] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8775.867544] LNet: Added LNI 192.168.202.152@tcp47 [8/256/0/180] [ 8778.473432] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8778.484670] LNet: Added LNI 192.168.202.152@tcp48 [8/256/0/180] [ 8781.120065] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8781.130838] LNet: Added LNI 192.168.202.152@tcp49 [8/256/0/180] [ 8783.615738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8783.628815] LNet: Added LNI 192.168.202.152@tcp50 [8/256/0/180] [ 8792.031749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8792.033726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.037171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.056709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.057186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.062165] Lustre: Skipped 1 previous similar message [ 8796.955600] Lustre: server umount lustre-MDT0000 complete [ 8798.134107] LustreError: 449277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774895311 with bad export cookie 107281823412875554 [ 8798.136580] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8798.139228] LustreError: 449277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8798.176585] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8798.181683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8798.183986] Lustre: Skipped 2 previous similar messages [ 8798.272439] Lustre: server umount lustre-MDT0001 complete [ 8804.889228] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8807.686211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8808.415323] LustreError: 442540:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d40119bf x1861112505118848/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8808.519687] LustreError: 445002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.525038] LustreError: 445002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8808.536946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.771946] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8812.468339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.592588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8813.840844] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8815.549591] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8816.643526] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8818.659861] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8823.777770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.778241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8823.779118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8823.780821] Lustre: Skipped 1 previous similar message [ 8828.897079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8828.898602] Lustre: Skipped 2 previous similar messages [ 8829.652194] Lustre: server umount lustre-OST0000 complete [ 8832.991880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8832.993818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.136070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.137824] Lustre: Skipped 4 previous similar messages [ 8845.279078] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8845.333905] Lustre: server umount lustre-MDT0000 complete [ 8846.539636] LustreError: 443261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774895359 with bad export cookie 107281823412877570 [ 8846.543061] LustreError: 443261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8846.543537] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.652893] Lustre: server umount lustre-MDT0001 complete [ 8850.113511] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8851.047598] Key type lgssc unregistered [ 8851.206285] LNet: 470925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8851.209091] LNetError: 470925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8851.210984] LNet: Removed LNI 192.168.202.152@tcp [ 8851.212084] LNet: Removed LNI 192.168.202.152@tcp10 [ 8851.213079] LNet: Removed LNI 192.168.202.152@tcp11 [ 8851.214204] LNet: Removed LNI 192.168.202.152@tcp12 [ 8851.215266] LNet: Removed LNI 192.168.202.152@tcp13 [ 8851.216357] LNet: Removed LNI 192.168.202.152@tcp14 [ 8851.217454] LNet: Removed LNI 192.168.202.152@tcp15 [ 8851.218577] LNet: Removed LNI 192.168.202.152@tcp16 [ 8851.219619] LNet: Removed LNI 192.168.202.152@tcp17 [ 8851.220820] LNet: Removed LNI 192.168.202.152@tcp18 [ 8851.222089] LNet: Removed LNI 192.168.202.152@tcp19 [ 8851.223350] LNet: Removed LNI 192.168.202.152@tcp20 [ 8851.224906] LNet: Removed LNI 192.168.202.152@tcp21 [ 8851.226837] LNet: Removed LNI 192.168.202.152@tcp22 [ 8851.228182] LNet: Removed LNI 192.168.202.152@tcp23 [ 8851.229797] LNet: Removed LNI 192.168.202.152@tcp24 [ 8851.231152] LNet: Removed LNI 192.168.202.152@tcp25 [ 8851.232248] LNet: Removed LNI 192.168.202.152@tcp26 [ 8851.233560] LNet: Removed LNI 192.168.202.152@tcp27 [ 8851.234683] LNet: Removed LNI 192.168.202.152@tcp28 [ 8851.236083] LNet: Removed LNI 192.168.202.152@tcp29 [ 8851.237268] LNet: Removed LNI 192.168.202.152@tcp30 [ 8851.238502] LNet: Removed LNI 192.168.202.152@tcp31 [ 8851.239827] LNet: Removed LNI 192.168.202.152@tcp32 [ 8851.241044] LNet: Removed LNI 192.168.202.152@tcp33 [ 8851.242268] LNet: Removed LNI 192.168.202.152@tcp34 [ 8851.243612] LNet: Removed LNI 192.168.202.152@tcp35 [ 8851.244694] LNet: Removed LNI 192.168.202.152@tcp36 [ 8851.245720] LNet: Removed LNI 192.168.202.152@tcp37 [ 8851.246779] LNet: Removed LNI 192.168.202.152@tcp38 [ 8851.248195] LNet: Removed LNI 192.168.202.152@tcp39 [ 8851.249312] LNet: Removed LNI 192.168.202.152@tcp40 [ 8851.250550] LNet: Removed LNI 192.168.202.152@tcp41 [ 8851.251552] LNet: Removed LNI 192.168.202.152@tcp42 [ 8851.252732] LNet: Removed LNI 192.168.202.152@tcp43 [ 8851.253791] LNet: Removed LNI 192.168.202.152@tcp44 [ 8851.254953] LNet: Removed LNI 192.168.202.152@tcp45 [ 8851.256514] LNet: Removed LNI 192.168.202.152@tcp46 [ 8851.257916] LNet: Removed LNI 192.168.202.152@tcp47 [ 8851.258899] LNet: Removed LNI 192.168.202.152@tcp48 [ 8851.260462] LNet: Removed LNI 192.168.202.152@tcp49 [ 8851.267035] LNet: Removed LNI 192.168.202.152@tcp50 [ 8851.571742] Key type .llcrypt unregistered [ 8851.573221] Key type ._llcrypt unregistered [ 8858.740557] Key type ._llcrypt registered [ 8858.741612] Key type .llcrypt registered [ 8858.777647] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8859.200545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8859.207895] alg: No test for adler32 (adler32-zlib) [ 8860.059650] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 8860.149300] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8861.727144] Key type lgssc registered [ 8862.073115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8864.507774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8864.514726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.668540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.951399] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8869.723911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.849532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8871.175745] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8873.024200] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8874.295531] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8876.584458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8876.704150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8878.509242] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8880.243667] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8890.023590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8894.027223] loop0: detected capacity change from 0 to 204800000 [ 8894.139255] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8894.142310] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8894.145533] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8896.121664] loop0: detected capacity change from 0 to 204800000 [ 8896.229831] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8896.233210] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8896.238339] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8897.948845] loop0: detected capacity change from 0 to 204800000 [ 8899.900282] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8899.957641] Lustre: fs15246-OST0000: new disk, initializing [ 8899.959627] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8899.979259] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.791697] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8905.490109] loop1: detected capacity change from 0 to 204800000 [ 8907.282088] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.317985] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8907.338765] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8907.374203] Lustre: fs15246-MDT0000: new disk, initializing [ 8907.423251] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.431705] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8907.473874] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8908.512653] Lustre: 476528:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000009b8afaf7 x1861112690669440/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1774895437 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8908.655143] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8917.479990] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8917.599259] LustreError: 476528:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8917.603082] LustreError: 476528:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8917.605558] LustreError: 476528:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8922.003973] Lustre: server umount fs15246-MDT0000 complete [ 8922.597362] LustreError: 472142:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.603568] LustreError: 472142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8925.047986] Lustre: server umount fs15246-OST0000 complete [ 8931.296725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8931.300325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.306850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8932.320267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8932.326056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.335194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.480943] Lustre: server umount lustre-OST0000 complete [ 8936.415915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8936.418905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.424577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8936.426944] Lustre: Skipped 1 previous similar message [ 8941.863723] Lustre: server umount lustre-MDT0000 complete [ 8943.064106] LustreError: 472127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774895456 with bad export cookie 1437975177474255645 [ 8943.066794] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8943.067050] LustreError: 472127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8943.176423] Lustre: server umount lustre-MDT0001 complete [ 8946.782615] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8947.761592] Key type lgssc unregistered [ 8947.891630] LNet: 479587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8947.896277] LNetError: 479587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8947.910583] LNet: Removed LNI 192.168.202.152@tcp [ 8948.207108] Key type .llcrypt unregistered [ 8948.208142] Key type ._llcrypt unregistered [ 8955.210524] Key type ._llcrypt registered [ 8955.211634] Key type .llcrypt registered [ 8955.247308] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8955.608078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8955.635717] alg: No test for adler32 (adler32-zlib) [ 8956.483367] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 8956.563388] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8958.143155] Key type lgssc registered [ 8958.604108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8970.664408] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8973.405451] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8984.372241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8984.377049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.465374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8985.476180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8985.512913] Lustre: lustre-MDT0000: new disk, initializing [ 8985.538188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.545882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8986.798560] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8990.845064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.897430] Lustre: 483970:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8990.923531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8990.926804] Lustre: Skipped 1 previous similar message [ 8990.968246] Lustre: lustre-MDT0001: new disk, initializing [ 8991.005091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8991.018459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8991.021690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8992.357838] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8995.414610] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8996.600274] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8998.724506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8998.825329] Lustre: lustre-OST0000: new disk, initializing [ 8998.827433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8998.854560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.620308] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9002.873572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9002.878514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9002.903816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9003.589870] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9004.870186] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9004.940114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9006.199368] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9006.266242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9008.101437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9008.101555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.103742] LustreError: Skipped 1 previous similar message [ 9008.110181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9008.113248] Lustre: Skipped 1 previous similar message [ 9013.009240] Lustre: server umount lustre-OST0000 complete [ 9016.799915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9016.802025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.805203] Lustre: Skipped 1 previous similar message [ 9016.806975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9018.336222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9018.336801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.390666] Lustre: server umount lustre-MDT0000 complete [ 9021.642310] LustreError: 483963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774895534 with bad export cookie 9920974936655750531 [ 9021.645593] LustreError: 483963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9021.645984] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9021.780562] Lustre: server umount lustre-MDT0001 complete [ 9029.390781] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9032.442630] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9042.505787] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8777 sec ========== 14:32:34 (1774895554) [ 9043.020899] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:32:35 (1774895555) === [ 9044.061420] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:32:36 (1774895556) === [ 9056.774265] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9057.674549] Key type lgssc unregistered [ 9057.803322] LNet: 492146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9057.805391] LNetError: 492146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9057.817240] LNet: Removed LNI 192.168.202.152@tcp [ 9058.093413] Key type .llcrypt unregistered [ 9058.094395] Key type ._llcrypt unregistered