[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 583258717 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003072] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008473] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011175] LSM: Security Framework initializing [ 0.013055] Yama: becoming mindful. [ 0.014000] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.023600] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028671] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030074] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031099] *** VALIDATE tmpfs *** [ 0.032306] *** VALIDATE proc *** [ 0.033117] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042295] debug: unmapping init [mem 0xffffffff8b259000-0xffffffff8b260fff] [ 0.045265] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054297] rcu: Hierarchical SRCU implementation. [ 0.056725] smp: Bringing up secondary CPUs ... [ 0.057655] x86: Booting SMP configuration: [ 0.058023] .... node #0, CPUs: #1 #2 #3 [ 0.063159] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.214492] node 0 deferred pages initialised in 145ms [ 0.219123] devtmpfs: initialized [ 0.221320] x86/mm: Memory block size: 128MB [ 0.225189] gcov: version magic: 0x41383552 [ 0.229338] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.230084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.231285] pinctrl core: initialized pinctrl subsystem [ 0.232171] [ 0.232660] ************************************************************* [ 0.233014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.234012] ** ** [ 0.235013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.236018] ** ** [ 0.237012] ** This means that this kernel is built to expose internal ** [ 0.238012] ** IOMMU data structures, which may compromise security on ** [ 0.239014] ** your system. ** [ 0.240017] ** ** [ 0.241013] ** If you see this message and you are not debugging the ** [ 0.242013] ** kernel, report this immediately to your vendor! ** [ 0.243013] ** ** [ 0.244013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245014] ************************************************************* [ 0.247000] NET: Registered protocol family 16 [ 0.250551] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.254100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.257086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.262207] cpuidle: using governor menu [ 0.264929] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268522] PCI: Using configuration type 1 for base access [ 0.270168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.281063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283121] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287108] cryptd: max_cpu_qlen set to 1000 [ 0.290328] ACPI: Added _OSI(Module Device) [ 0.292021] ACPI: Added _OSI(Processor Device) [ 0.293017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.295016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299380] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.307366] ACPI: Interpreter enabled [ 0.308079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.310018] ACPI: Using IOAPIC for interrupt routing [ 0.312167] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.314463] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.326524] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.328065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331092] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.334104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.339562] acpiphp: Slot [2] registered [ 0.341145] acpiphp: Slot [5] registered [ 0.342170] acpiphp: Slot [6] registered [ 0.344108] acpiphp: Slot [7] registered [ 0.346224] acpiphp: Slot [8] registered [ 0.347179] acpiphp: Slot [9] registered [ 0.349190] acpiphp: Slot [10] registered [ 0.351171] acpiphp: Slot [3] registered [ 0.352143] acpiphp: Slot [4] registered [ 0.354141] acpiphp: Slot [11] registered [ 0.355142] acpiphp: Slot [12] registered [ 0.357275] acpiphp: Slot [13] registered [ 0.359221] acpiphp: Slot [14] registered [ 0.361176] acpiphp: Slot [15] registered [ 0.362135] acpiphp: Slot [16] registered [ 0.364150] acpiphp: Slot [17] registered [ 0.366161] acpiphp: Slot [18] registered [ 0.368181] acpiphp: Slot [19] registered [ 0.369155] acpiphp: Slot [20] registered [ 0.371171] acpiphp: Slot [21] registered [ 0.373136] acpiphp: Slot [22] registered [ 0.374161] acpiphp: Slot [23] registered [ 0.376141] acpiphp: Slot [24] registered [ 0.377147] acpiphp: Slot [25] registered [ 0.378131] acpiphp: Slot [26] registered [ 0.380143] acpiphp: Slot [27] registered [ 0.381139] acpiphp: Slot [28] registered [ 0.383196] acpiphp: Slot [29] registered [ 0.384126] acpiphp: Slot [30] registered [ 0.386154] acpiphp: Slot [31] registered [ 0.388113] PCI host bridge to bus 0000:00 [ 0.389027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.396032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.399036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.402041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.404223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.407333] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.410630] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.421799] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426923] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.434029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.437708] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.440877] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.443060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447133] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.452024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.464026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.469021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.476412] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.482018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.489019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.503026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.514135] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.531026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.546028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.555874] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.561014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.600853] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.609026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.615026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.664807] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.675026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683073] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.700025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711125] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.730025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.736024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.751027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.765106] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.770133] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.773459] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.777496] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.779275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.786037] iommu: Default domain type: Passthrough [ 0.787577] SCSI subsystem initialized [ 0.789174] ACPI: bus type USB registered [ 0.790140] usbcore: registered new interface driver usbfs [ 0.792158] usbcore: registered new interface driver hub [ 0.794126] usbcore: registered new device driver usb [ 0.796235] pps_core: LinuxPPS API ver. 1 registered [ 0.798013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.800086] PTP clock support registered [ 0.803116] EDAC MC: Ver: 3.0.0 [ 0.805236] PCI: Using ACPI for IRQ routing [ 0.807199] NetLabel: Initializing [ 0.808014] NetLabel: domain hash size = 128 [ 0.809015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.810129] NetLabel: unlabeled traffic allowed by default [ 0.813145] vgaarb: loaded [ 0.815463] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.817018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.825062] clocksource: Switched to clocksource kvm-clock [ 0.944405] VFS: Disk quotas dquot_6.6.0 [ 0.946147] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.949024] *** VALIDATE ramfs *** [ 0.950433] *** VALIDATE hugetlbfs *** [ 0.952218] pnp: PnP ACPI init [ 0.954894] pnp: PnP ACPI: found 6 devices [ 1.004203] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.007548] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.010141] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.012773] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.015232] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.017705] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.021824] NET: Registered protocol family 2 [ 1.024505] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.029458] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.032903] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.038066] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.041382] TCP: Hash tables configured (established 65536 bind 65536) [ 1.044050] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.046155] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049192] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.052119] NET: Registered protocol family 1 [ 1.055480] RPC: Registered named UNIX socket transport module. [ 1.057493] RPC: Registered udp transport module. [ 1.059369] RPC: Registered tcp transport module. [ 1.060891] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.063581] NET: Registered protocol family 44 [ 1.065443] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067723] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.070163] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072489] PCI: CLS 0 bytes, default 64 [ 1.074235] Unpacking initramfs... [ 2.568849] debug: unmapping init [mem 0xffff9b27fcc54000-0xffff9b27fffbffff] [ 2.572881] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.575135] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.579632] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.128843] Initialise system trusted keyrings [ 3.132930] Key type blacklist registered [ 3.136915] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.149402] zbud: loaded [ 3.153204] *** VALIDATE nfs *** [ 3.154520] *** VALIDATE nfs4 *** [ 3.156950] pstore: using deflate compression [ 3.160546] Platform Keyring initialized [ 3.288794] NET: Registered protocol family 38 [ 3.290937] Key type asymmetric registered [ 3.292808] Asymmetric key parser 'x509' registered [ 3.295194] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.299025] io scheduler mq-deadline registered [ 3.301220] io scheduler kyber registered [ 3.303202] io scheduler bfq registered [ 3.305300] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.308801] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.311681] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.314455] ACPI: Power Button [PWRF] [ 3.320256] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.327755] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.386817] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.395792] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.423594] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.452297] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.482078] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.486989] Non-volatile memory driver v1.3 [ 3.488483] Linux agpgart interface v0.103 [ 3.520842] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.524834] vda: detected capacity change from 0 to 69914624 [ 3.543264] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.545978] vdb: detected capacity change from 0 to 1073741824 [ 3.564773] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.567839] vdc: detected capacity change from 0 to 2621440000 [ 3.584091] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.587102] vdd: detected capacity change from 0 to 2621440000 [ 3.612157] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.615040] vde: detected capacity change from 0 to 4294967296 [ 3.632365] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.636241] vdf: detected capacity change from 0 to 4294967296 [ 3.649507] libphy: Fixed MDIO Bus: probed [ 3.656162] usbcore: registered new interface driver usbserial_generic [ 3.659762] usbserial: USB Serial support registered for generic [ 3.661634] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.665866] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.667616] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.670195] mousedev: PS/2 mouse device common for all mice [ 3.673318] rtc_cmos 00:05: RTC can wake from S4 [ 3.676300] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.680310] rtc_cmos 00:05: registered as rtc0 [ 3.683693] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.684190] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.690476] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.691690] intel_pstate: CPU model not supported [ 3.701541] hid: raw HID events driver (C) Jiri Kosina [ 3.704049] usbcore: registered new interface driver usbhid [ 3.706230] usbhid: USB HID core driver [ 3.708022] drop_monitor: Initializing network drop monitor service [ 3.710575] Initializing XFRM netlink socket [ 3.712849] NET: Registered protocol family 10 [ 3.716053] Segment Routing with IPv6 [ 3.717492] NET: Registered protocol family 17 [ 3.720313] mpls_gso: MPLS GSO support [ 3.725962] RAS: Correctable Errors collector initialized. [ 3.728581] AVX version of gcm_enc/dec engaged. [ 3.730173] AES CTR mode by8 optimization enabled [ 3.818875] sched_clock: Marking stable (3818836788, 0)->(4777873916, -959037128) [ 3.823613] registered taskstats version 1 [ 3.825727] Loading compiled-in X.509 certificates [ 3.827772] zswap: loaded using pool lzo/zbud [ 3.858420] Key type big_key registered [ 3.871889] Key type encrypted registered [ 3.873569] ima: No TPM chip found, activating TPM-bypass! [ 3.875738] ima: Allocated hash algorithm: sha1 [ 3.877745] ima: No architecture policies found [ 3.879839] evm: Initialising EVM extended attributes: [ 3.882018] evm: security.selinux [ 3.883322] evm: security.ima [ 3.884600] evm: security.capability [ 3.886520] evm: HMAC attrs: 0x1 [ 3.889837] rtc_cmos 00:05: setting system clock to 2026-06-12 00:40:27 UTC (1781224827) [ 3.896727] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 3.900518] debug: unmapping init [mem 0xffffffff8af82000-0xffffffff8b258fff] [ 3.910084] Write protecting the kernel read-only data: 28672k [ 3.914061] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 3.916794] debug: unmapping init [mem 0xffffffff89f14000-0xffffffff89ffffff] [ 3.958325] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.966060] systemd[1]: Detected virtualization kvm. [ 3.968178] systemd[1]: Detected architecture x86-64. [ 3.970632] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.000699] systemd[1]: No hostname configured. [ 4.002669] systemd[1]: Set hostname to . [ 4.005339] random: systemd: uninitialized urandom read (16 bytes read) [ 4.008505] systemd[1]: Initializing machine ID from random generator. [ 4.100342] random: ln: uninitialized urandom read (6 bytes read) [ 4.209811] random: systemd: uninitialized urandom read (16 bytes read) [ 4.212876] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.221718] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.234132] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.149446] device-mapper: uevent: version 1.0.3 [ 5.151123] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.198517] random: fast init done [ 6.264643] virtio_net virtio0 ens2: renamed from eth0 [ 6.376114] scsi host0: ata_piix [ 6.561136] scsi host1: ata_piix [ 6.562905] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.564811] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.262108] dracut-initqueue[581]: RTNETLINK answers: File exists [ 11.392596] random: crng init done [ 11.394196] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.807797] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.847722] printk: systemd: 24 output lines suppressed due to ratelimiting [ 13.061878] SELinux: Disabled at runtime. [ 13.128387] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.135279] systemd[1]: Detected virtualization kvm. [ 13.137446] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.570417] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.574208] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.578644] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.583257] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.586727] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.594167] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.598160] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ 13.661968] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.096651] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.453841] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.483437] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.615471] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.624350] EDAC sbridge: Ver: 1.1.2 [ 16.286825] Key type dns_resolver registered [ 16.573661] NFS: Registering the id_resolver key type [ 16.575643] Key type id_resolver registered [ 16.577099] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg117-server login: [ 29.783618] hrtimer: interrupt took 2934736 ns [ 78.389190] libcfs: loading out-of-tree module taints kernel. [ 78.450558] Key type ._llcrypt registered [ 78.452740] Key type .llcrypt registered [ 78.590331] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 100.775823] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 102.844406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.881856] alg: No test for adler32 (adler32-zlib) [ 104.864170] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 106.075676] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 107.921638] Key type lgssc registered [ 110.039528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.950745] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.286760] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 196.382656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.421735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.678934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.724281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.823141] Lustre: lustre-MDT0000: new disk, initializing [ 197.924943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.942652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.366693] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.842734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.990949] Lustre: 6506:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.018467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.021577] Lustre: Skipped 1 previous similar message [ 220.087541] Lustre: lustre-MDT0001: new disk, initializing [ 220.179577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.227196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.239505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.553381] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.470866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.661645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.054192] Lustre: lustre-OST0000: new disk, initializing [ 240.056344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.076892] Lustre: 8409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.218912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.551306] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.403885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.414704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.534161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.248900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.358676] Lustre: lustre-OST0001: new disk, initializing [ 264.370970] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.380829] Lustre: 9466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.477711] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 270.801439] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.411827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.421513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 271.485798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.564107] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.393368] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.393724] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.602450] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing yml_node [ 315.491027] Lustre: DEBUG MARKER: Client: 2.17.53.70 [ 318.002729] Lustre: DEBUG MARKER: MDS: 2.17.53.70 [ 320.835760] Lustre: DEBUG MARKER: OSS: 2.17.53.70 [ 322.689880] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 20:45:44 EDT 2026 [ 340.638599] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 342.199196] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.375840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.386593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.507819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.513038] Lustre: Skipped 6 previous similar messages [ 366.766679] Lustre: server umount lustre-MDT0000 complete [ 371.924374] LustreError: 8410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225195 with bad export cookie 18393255551552402564 [ 371.927351] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.933882] LustreError: 8410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 373.732790] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.736729] LustreError: 12205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.746500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.756481] Lustre: Skipped 4 previous similar messages [ 373.799501] LustreError: 12205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 376.750816] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.751455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.779508] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.819226] Lustre: Skipped 2 previous similar messages [ 378.430833] Lustre: server umount lustre-MDT0001 complete [ 383.935634] Lustre: server umount lustre-OST0000 complete [ 388.749567] Lustre: server umount lustre-OST0001 complete [ 395.611163] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 405.504406] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 445.363809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.735361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.793918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.896980] Lustre: lustre-MDT0000: new disk, initializing [ 446.044833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.069991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.989489] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 462.762243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.913679] Lustre: 14985:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.982207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.990160] Lustre: Skipped 1 previous similar message [ 463.063563] Lustre: lustre-MDT0001: new disk, initializing [ 463.167024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.200970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.213680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.083659] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 476.632838] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.351328] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 488.731918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.154650] Lustre: lustre-OST0000: new disk, initializing [ 489.162155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 489.178323] Lustre: 16726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 489.349911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.210378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 491.239719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 491.370826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.399311] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 504.770281] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 510.265755] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 510.492466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 516.964795] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.270753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 522.217157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.221544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.232367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.247996] Lustre: Skipped 1 previous similar message [ 522.259859] LustreError: Skipped 1 previous similar message [ 525.847883] Lustre: server umount lustre-OST0000 complete [ 532.452893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.477955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 532.484173] Lustre: Skipped 2 previous similar messages [ 536.384479] Lustre: server umount lustre-MDT0000 complete [ 537.568211] LustreError: 15849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.595885] LustreError: 15849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 540.487139] LustreError: 15762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225364 with bad export cookie 18393255551552404097 [ 540.499668] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.502336] LustreError: 15762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 541.029410] Lustre: server umount lustre-MDT0001 complete [ 545.219669] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:49:26 (1781225366) [ 548.435247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 550.350485] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:49:32 (1781225372) [ 558.146151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.798324] LustreError: 18988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.915458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.203057] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.293819] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 574.435030] LustreError: 18988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.464378] LustreError: 18988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 575.175474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.904952] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 586.956944] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 591.678859] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 599.036640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 605.412242] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 610.362977] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 618.978027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.003856] Lustre: Skipped 1 previous similar message [ 619.007336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.015653] Lustre: Skipped 1 previous similar message [ 624.429472] Lustre: server umount lustre-OST0000 complete [ 629.217024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.241783] Lustre: Skipped 2 previous similar messages [ 635.274953] Lustre: server umount lustre-MDT0000 complete [ 639.123220] LustreError: 19708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225462 with bad export cookie 18393255551552404979 [ 639.129850] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.134649] LustreError: 19708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 639.463602] Lustre: server umount lustre-MDT0001 complete [ 650.815638] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 654.374394] Key type lgssc unregistered [ 654.685787] LNet: 22165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.691970] LNetError: 22165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.704932] LNet: Removed LNI 192.168.201.117@tcp [ 655.706555] Key type .llcrypt unregistered [ 655.712787] Key type ._llcrypt unregistered [ 680.661720] Key type ._llcrypt registered [ 680.664686] Key type .llcrypt registered [ 680.842821] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 682.737073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.802184] alg: No test for adler32 (adler32-zlib) [ 683.982542] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 684.306947] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 686.039158] Key type lgssc registered [ 687.282426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.997693] loop: module loaded [ 695.094187] loop0: detected capacity change from 0 to 204800000 [ 695.419602] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.443305] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.746034] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.539872] loop0: detected capacity change from 0 to 204800000 [ 704.106479] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.129655] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.151265] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 711.503088] loop0: detected capacity change from 0 to 204800000 [ 720.518715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 720.569684] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.147268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 722.302252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 722.465660] Lustre: lustre-MDT0000: new disk, initializing [ 722.635572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.667237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 727.325824] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 738.108131] loop1: detected capacity change from 0 to 204800000 [ 744.215504] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.533947] Lustre: lustre-OST07c6: new disk, initializing [ 744.538234] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 744.542236] Lustre: Skipped 1 previous similar message [ 744.544392] Lustre: 25331:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 744.710066] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 751.160684] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 753.794176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 753.806954] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 753.881227] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 757.349880] Lustre: 25830:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 757.364807] Lustre: 25830:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 757.374362] Lustre: 25830:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 757.383900] Lustre: 25830:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 757.394034] Lustre: 25830:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 757.401225] Lustre: 25830:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 771.609320] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 771.622521] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 771.630525] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.637869] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 771.647284] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 771.660020] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.579350] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 776.587814] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.597357] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 776.603427] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.609835] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.615018] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.622837] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 776.628698] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.642900] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.649335] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.654217] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.659935] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.933475] Lustre: 24334:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 779.948488] Lustre: 24334:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 779.959668] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 779.968533] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.983522] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.002346] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.014277] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 780.023663] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.036787] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 780.047935] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.058895] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.068607] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.470931] Lustre: 25830:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 785.483822] Lustre: 25830:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.495817] Lustre: 25830:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.504639] Lustre: 25830:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.518876] Lustre: 25830:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.522897] Lustre: 25830:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.527226] Lustre: 25830:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 785.532304] Lustre: 25830:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.536752] Lustre: 25830:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.541952] Lustre: 25830:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.547392] Lustre: 25830:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.555418] Lustre: 25830:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.540598] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 793.549754] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 793.556405] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.575396] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 793.587115] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.598288] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 793.602765] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.608529] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 793.614989] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 793.618056] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 793.627411] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.633022] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.088280] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 810.099509] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 810.105495] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 810.111852] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 810.116739] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 810.121746] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 810.126489] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 810.131258] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 810.136784] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 810.143251] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 810.148029] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.152986] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 850.916952] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.956806] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 853.493699] Lustre: server umount lustre-OST07c6 complete [ 866.843126] Lustre: server umount lustre-MDT0000 complete [ 884.150776] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 888.303344] Key type lgssc unregistered [ 888.806773] LNet: 27853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.822655] LNetError: 27853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.858559] LNet: Removed LNI 192.168.201.117@tcp [ 889.784471] Key type .llcrypt unregistered [ 889.786571] Key type ._llcrypt unregistered [ 913.464955] Key type ._llcrypt registered [ 913.470274] Key type .llcrypt registered [ 913.590819] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 915.122148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.366427] alg: No test for adler32 (adler32-zlib) [ 916.520430] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 916.745399] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 918.455191] Key type lgssc registered [ 920.020837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 928.682817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 928.702982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.421542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.731583] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 940.000628] LustreError: 29109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 940.025516] LustreError: 29109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 944.203836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.787695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.931717] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 954.201280] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 957.287169] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 965.930286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 966.425481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.591814] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 974.659442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 978.598690] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1000.417925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.420721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.457047] Lustre: Skipped 1 previous similar message [ 1000.489253] Lustre: Skipped 1 previous similar message [ 1004.129308] Lustre: server umount lustre-OST0000 complete [ 1010.661720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.666181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.695076] Lustre: Skipped 1 previous similar message [ 1010.732396] Lustre: Skipped 1 previous similar message [ 1015.778807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.790840] Lustre: Skipped 1 previous similar message [ 1016.379818] Lustre: server umount lustre-MDT0000 complete [ 1020.790460] LustreError: 29089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225844 with bad export cookie 3845503529211864646 [ 1020.805668] LustreError: 29089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1020.812205] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.574972] Lustre: server umount lustre-MDT0001 complete [ 1035.082687] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1039.299500] Key type lgssc unregistered [ 1039.784349] LNet: 32491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.802968] LNetError: 32491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.879268] LNet: Removed LNI 192.168.201.117@tcp [ 1042.263752] Key type .llcrypt unregistered [ 1042.265977] Key type ._llcrypt unregistered [ 1065.050686] Key type ._llcrypt registered [ 1065.052635] Key type .llcrypt registered [ 1065.183159] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1066.415619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.673108] alg: No test for adler32 (adler32-zlib) [ 1067.712448] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1067.961978] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1069.672397] Key type lgssc registered [ 1070.920704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1079.010631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1079.035054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.711481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.919811] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1091.049075] LustreError: 33747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.067416] LustreError: 33747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1093.292198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.688429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.460542] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1104.771071] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1108.671649] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1115.502679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1115.737947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.004709] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1126.014781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1129.667883] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.728756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.753427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.758788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.771567] Lustre: Skipped 1 previous similar message [ 1141.787991] Lustre: Skipped 1 previous similar message [ 1146.259783] Lustre: server umount lustre-OST0000 complete [ 1151.975598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.984167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.996953] Lustre: Skipped 1 previous similar message [ 1157.115608] LustreError: 33748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.609590] Lustre: server umount lustre-MDT0000 complete [ 1162.133901] LustreError: 35363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225985 with bad export cookie 4850488934101533207 [ 1162.138521] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.151744] LustreError: 35363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.823894] Lustre: server umount lustre-MDT0001 complete [ 1176.939894] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1180.659619] Key type lgssc unregistered [ 1181.066910] LNet: 36926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.076303] LNetError: 36926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.106754] LNet: Removed LNI 192.168.201.117@tcp [ 1182.126466] Key type .llcrypt unregistered [ 1182.128436] Key type ._llcrypt unregistered [ 1206.963897] Key type ._llcrypt registered [ 1206.967665] Key type .llcrypt registered [ 1207.136475] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1209.008758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1209.034547] alg: No test for adler32 (adler32-zlib) [ 1210.157881] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1210.516775] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1212.283105] Key type lgssc registered [ 1213.889326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1223.500142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1223.549511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.034768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.639329] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1234.416438] LustreError: 38331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.442835] LustreError: 38331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1239.525032] LustreError: 38330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.577019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.208610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1247.207480] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1253.622869] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.771362] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1264.787527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.251760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.388882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1271.426229] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1277.498542] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1285.789076] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 1286.121511] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.153968] Lustre: Skipped 1 previous similar message [ 1290.838965] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 1290.848826] Lustre: Skipped 3 previous similar messages [ 1291.779864] Lustre: server umount lustre-MDT0000 complete [ 1295.958746] LustreError: 38326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.094985] LustreError: 38310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226119 with bad export cookie 17974478277940022477 [ 1296.096866] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.110531] LustreError: 38310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1296.354421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.371523] Lustre: Skipped 1 previous similar message [ 1296.376858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1296.389078] Lustre: Skipped 3 previous similar messages [ 1301.086718] LustreError: 38325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.118792] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [ 1301.128557] LustreError: 38325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1302.710169] Lustre: server umount lustre-MDT0001 complete [ 1323.841946] Lustre: server umount lustre-OST0000 complete [ 1341.163812] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1344.605367] Key type lgssc unregistered [ 1344.981704] LNet: 41849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.987395] LNetError: 41849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.004524] LNet: Removed LNI 192.168.201.117@tcp [ 1346.143263] Key type .llcrypt unregistered [ 1346.147604] Key type ._llcrypt unregistered [ 1375.890364] Key type ._llcrypt registered [ 1375.894122] Key type .llcrypt registered [ 1376.099226] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1377.926077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.089080] alg: No test for adler32 (adler32-zlib) [ 1379.326383] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1379.497757] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1381.232043] Key type lgssc registered [ 1383.215236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.239949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.257670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.067621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.508973] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1405.415438] LustreError: 43251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.443965] LustreError: 43251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1408.559173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.092462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1414.182792] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1419.530319] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1422.867394] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1429.962825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.340900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.020569] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1440.623419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1442.065397] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1449.959804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.968306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.979273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.980458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.982184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.990325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.011943] Lustre: Skipped 2 previous similar messages [ 1455.192134] LustreError: 44850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.404292] Lustre: server umount lustre-OST0000 complete [ 1475.552221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1475.561632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.574890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.582034] Lustre: Skipped 1 previous similar message [ 1475.595286] Lustre: Skipped 1 previous similar message [ 1481.697494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.702515] Lustre: Skipped 3 previous similar messages [ 1486.817064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.327098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1487.599443] Lustre: server umount lustre-MDT0000 complete [ 1491.273188] LustreError: 43971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226314 with bad export cookie 12503303622713594008 [ 1491.275942] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.282437] LustreError: 43971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1497.664099] Lustre: server umount lustre-MDT0001 complete [ 1508.174758] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1511.127449] Key type lgssc unregistered [ 1511.348820] LNet: 46607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.358333] LNetError: 46607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.369908] LNet: Removed LNI 192.168.201.117@tcp [ 1512.198092] Key type .llcrypt unregistered [ 1512.199827] Key type ._llcrypt unregistered [ 1537.225555] Key type ._llcrypt registered [ 1537.228242] Key type .llcrypt registered [ 1537.378541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1538.753900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1538.780937] alg: No test for adler32 (adler32-zlib) [ 1539.918962] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1540.193343] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1541.871210] Key type lgssc registered [ 1543.101418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1552.310740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1552.368892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.124670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.851818] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1563.620712] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.638038] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1568.746982] LustreError: 48007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.264121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.612218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1574.670940] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1582.122934] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1585.989694] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1593.445850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1593.959718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.143907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1599.547644] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1605.169448] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1611.298714] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1615.500994] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1628.547773] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1121h07m28s [ 1630.668846] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1633.419657] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 1634.456933] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 1635.296657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.316769] Lustre: Skipped 1 previous similar message [ 1636.386415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.406783] Lustre: Skipped 5 previous similar messages [ 1639.510243] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 1639.769165] Lustre: server umount lustre-MDT0000 complete [ 1642.466880] LustreError: 48007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.502905] LustreError: 48007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1643.797966] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1645.697863] LustreError: 48000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.205728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.303048] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.456097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1650.495957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1651.549901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1653.973823] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1657.365654] Lustre: DEBUG MARKER: Wait for df (36101) ... [ 1658.962356] Lustre: DEBUG MARKER: done [ 1666.024514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.035151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.057461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1667.043607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.280533] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1677.295640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1677.306009] Lustre: Skipped 5 previous similar messages [ 1677.554674] Lustre: server umount lustre-OST0000 complete [ 1682.403937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.405414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1688.023340] Lustre: server umount lustre-MDT0000 complete [ 1688.566361] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.597822] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1691.652063] LustreError: 47988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226515 with bad export cookie 6459451778534178945 [ 1691.665721] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1691.666863] LustreError: 47988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1692.132099] Lustre: server umount lustre-MDT0001 complete [ 1701.909150] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1704.533244] Key type lgssc unregistered [ 1704.816889] LNet: 52557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.826731] LNetError: 52557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.874077] LNet: Removed LNI 192.168.201.117@tcp [ 1705.694733] Key type .llcrypt unregistered [ 1705.697078] Key type ._llcrypt unregistered [ 1745.004534] Key type ._llcrypt registered [ 1745.006975] Key type .llcrypt registered [ 1745.124880] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1746.730617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.769084] alg: No test for adler32 (adler32-zlib) [ 1747.847240] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1748.100820] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 1749.831174] Key type lgssc registered [ 1751.215810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.898635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.925867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.280231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1760.303360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1760.524246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.429504] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1775.344962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.703450] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1775.769679] Lustre: 54754:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1776.175960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1781.021905] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1786.625248] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1790.502313] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1797.622491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1797.793562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1797.994178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.050319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1806.607652] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 1814.824911] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1821.470492] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1121h10m43s [ 1824.159700] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1833.094895] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1836.208325] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1836.284373] LustreError: 56962:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1856.690802] Lustre: lustre-MDT0000: Client a6246750-771c-4c4d-ae7d-42e9b04a9237 (at 192.168.201.17@tcp) reconnecting [ 1876.311138] LustreError: 56962:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1878.983716] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1881.577747] Lustre: DEBUG MARKER: done [ 1891.296698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.300843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.311275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.316992] Lustre: Skipped 1 previous similar message [ 1896.419405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.424670] Lustre: Skipped 2 previous similar messages [ 1896.898478] Lustre: server umount lustre-OST0000 complete [ 1901.538052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.543325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.558361] Lustre: Skipped 1 previous similar message [ 1901.574635] Lustre: Skipped 1 previous similar message [ 1907.682262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.691371] Lustre: Skipped 3 previous similar messages [ 1913.828602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.844706] Lustre: Skipped 1 previous similar message [ 1915.361057] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.252157] Lustre: server umount lustre-MDT0000 complete [ 1921.628547] LustreError: 54746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226745 with bad export cookie 14404648202003752482 [ 1921.645160] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1921.648717] LustreError: 54746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1922.306662] Lustre: server umount lustre-MDT0001 complete [ 1936.254655] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 1939.747323] Key type lgssc unregistered [ 1940.051968] LNet: 58733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1940.061662] LNetError: 58733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.084622] LNet: Removed LNI 192.168.201.117@tcp [ 1941.114986] Key type .llcrypt unregistered [ 1941.116716] Key type ._llcrypt unregistered [ 1995.736482] Key type ._llcrypt registered [ 1995.741447] Key type .llcrypt registered [ 1995.912971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 1997.295958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.350958] alg: No test for adler32 (adler32-zlib) [ 1998.339243] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 1998.550855] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2000.240568] Key type lgssc registered [ 2001.347829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2005.787692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2005.813755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.128718] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2007.144036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2007.281750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.100899] Lustre: Failing over lustre-MDT0000 [ 2008.361723] Lustre: server umount lustre-MDT0000 complete [ 2015.854961] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:13:57 (1781226837) [ 2022.411538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.886648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.175890] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2036.074128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.228365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2036.274465] Lustre: 61513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2036.568813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.950400] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2046.818937] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2050.199434] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2056.782188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.960346] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2057.153547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.468921] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2066.633147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2067.681509] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2073.069083] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2082.295849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.311689] Lustre: Skipped 1 previous similar message [ 2082.311896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.329776] Lustre: Skipped 2 previous similar messages [ 2086.645628] Lustre: server umount lustre-MDT0000 complete [ 2090.989686] LustreError: 61508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226914 with bad export cookie 12251339588476973034 [ 2091.011155] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.411740] Lustre: server umount lustre-MDT0001 complete [ 2095.114921] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2099.607556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2108.747401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.898087] Lustre: 60436:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226916/real 1781226916] req@0000000084fa9c58 x1867751696143744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2108.937193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.954384] Lustre: Skipped 1 previous similar message [ 2113.056640] Lustre: 60436:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226920/real 1781226920] req@00000000eb7f36c8 x1867751696143872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.695876] LustreError: 63137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.728403] LustreError: 63137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2116.826975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.983121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2117.791278] Lustre: 60438:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226924/real 1781226924] req@00000000f7d3740f x1867751696144256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.822732] Lustre: 60438:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2118.821694] LustreError: 63138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.845438] LustreError: 64944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.873826] LustreError: 64944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2121.891431] Lustre: 60437:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226929/real 1781226929] req@00000000cc362c41 x1867751696144384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2123.162993] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2125.282513] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.308068] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2130.402568] LustreError: 64945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.435408] LustreError: 64945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2133.807108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.309523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.420326] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2139.063618] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2145.614883] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.303640] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2164.216232] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.250463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.265479] Lustre: Skipped 1 previous similar message [ 2167.243648] Lustre: server umount lustre-MDT0000 complete [ 2169.316295] LustreError: 64944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.350801] LustreError: 64944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2171.705787] LustreError: 64086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226995 with bad export cookie 12251339588476975316 [ 2171.708268] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.716884] LustreError: 64086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2172.046891] Lustre: server umount lustre-MDT0001 complete [ 2178.422336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2190.075822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.815384] Lustre: 60436:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226998/real 1781226998] req@0000000098f1ef8b x1867751696172928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2190.860311] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.878982] Lustre: Skipped 2 previous similar messages [ 2197.339442] LustreError: 63138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.441171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.493784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2201.714472] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2202.527423] Lustre: 60436:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227010/real 1781227010] req@00000000912442c5 x1867751696173568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2202.585432] Lustre: 60436:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2211.067177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.443875] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2216.457031] Lustre: Skipped 1 previous similar message [ 2216.720395] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2219.595204] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2224.575875] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2228.994215] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2234.865210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.866509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2234.877405] Lustre: Skipped 1 previous similar message [ 2234.897958] Lustre: Skipped 2 previous similar messages [ 2235.094514] Lustre: server umount lustre-MDT0000 complete [ 2238.892500] LustreError: 67402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.923345] LustreError: 67402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2238.946983] LustreError: 61506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227062 with bad export cookie 12251339588476976695 [ 2238.951643] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.972534] LustreError: 61506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2239.453161] Lustre: server umount lustre-MDT0001 complete [ 2246.313451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2250.262630] Lustre: DEBUG MARKER: files compared the same [ 2262.524351] Lustre: server umount lustre-OST0000 complete [ 2280.179122] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2282.997920] Key type lgssc unregistered [ 2283.299375] LNet: 70433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.314857] LNetError: 70433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.335369] LNet: Removed LNI 192.168.201.117@tcp [ 2284.090755] Key type .llcrypt unregistered [ 2284.093386] Key type ._llcrypt unregistered [ 2307.311598] Key type ._llcrypt registered [ 2307.315837] Key type .llcrypt registered [ 2307.471824] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2308.939967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.990392] alg: No test for adler32 (adler32-zlib) [ 2310.090408] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2310.383571] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2312.031926] Key type lgssc registered [ 2313.826242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.631860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.650609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.391350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.717402] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug +malloc all [ 2333.665738] LustreError: 71834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.695037] LustreError: 71834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2338.796705] LustreError: 71833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.687592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.193464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.604391] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug +malloc all [ 2353.207893] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2357.670703] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2366.073546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2366.535668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.572958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2374.095356] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug +malloc all [ 2380.894723] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2396.147159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.147888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.164235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.557596] Lustre: server umount lustre-OST0000 complete [ 2406.387386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.402812] Lustre: Skipped 1 previous similar message [ 2406.418982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.434726] Lustre: Skipped 2 previous similar messages [ 2411.488691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2411.502887] Lustre: Skipped 1 previous similar message [ 2415.585029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.592929] Lustre: Skipped 1 previous similar message [ 2419.167148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2419.446198] Lustre: server umount lustre-MDT0000 complete [ 2420.708020] LustreError: 73042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.724445] LustreError: 73042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2423.094099] LustreError: 71813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227246 with bad export cookie 10322560441759085143 [ 2423.095205] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.098838] LustreError: 71813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2423.413513] Lustre: server umount lustre-MDT0001 complete [ 2435.355790] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2439.420206] Key type lgssc unregistered [ 2439.708638] LNet: 75034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.729181] LNetError: 75034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.815424] LNet: Removed LNI 192.168.201.117@tcp [ 2441.896371] Key type .llcrypt unregistered [ 2441.900559] Key type ._llcrypt unregistered [ 2469.368378] Key type ._llcrypt registered [ 2469.370414] Key type .llcrypt registered [ 2469.510358] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2471.062348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.111073] alg: No test for adler32 (adler32-zlib) [ 2472.239229] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2472.484867] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2474.200351] Key type lgssc registered [ 2475.564802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.477683] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2484.498180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.128502] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2505.250348] LustreError: 76409:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [ 2531.935911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.498607] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2544.770825] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2556.875739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.071518] Lustre: *** cfs_fail_loc=706, val=0*** [ 2557.938895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.083763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2563.055320] LustreError: 76434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.085733] LustreError: 76434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2563.919283] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2567.159243] LustreError: 77382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.206623] LustreError: 77382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2572.257488] LustreError: 77381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.279225] LustreError: 77381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2573.376256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.707302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.500904] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2584.678310] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2588.538188] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2593.761090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.769280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.798598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.274510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.298294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.318766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.332563] Lustre: Skipped 1 previous similar message [ 2599.407192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.523492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.532482] Lustre: Skipped 2 previous similar messages [ 2605.023195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2605.133545] Lustre: server umount lustre-OST0000 complete [ 2609.635461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.651475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.227510] Lustre: server umount lustre-MDT0000 complete [ 2619.004350] LustreError: 76416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227442 with bad export cookie 9323889146764498803 [ 2619.008740] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.020231] LustreError: 76416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2619.463601] Lustre: server umount lustre-MDT0001 complete [ 2631.465201] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 2634.143797] Key type lgssc unregistered [ 2634.411462] LNet: 79670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.427610] LNetError: 79670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2634.460155] LNet: Removed LNI 192.168.201.117@tcp [ 2635.342891] Key type .llcrypt unregistered [ 2635.346250] Key type ._llcrypt unregistered [ 2665.021106] Key type ._llcrypt registered [ 2665.025156] Key type .llcrypt registered [ 2665.246220] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2667.015511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2667.041219] alg: No test for adler32 (adler32-zlib) [ 2668.189177] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2668.513465] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 2670.343161] Key type lgssc registered [ 2671.649306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2681.230563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2681.253167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.952324] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2694.738283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.346706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2699.900985] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2700.771619] LustreError: 81752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.796642] LustreError: 81752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2704.866176] LustreError: 81756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.889586] LustreError: 81756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2709.984465] LustreError: 81755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.012656] LustreError: 81755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2710.282760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.802371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.129527] LustreError: 81756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.157179] LustreError: 81756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2716.829725] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2720.226361] LustreError: 81755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.249062] LustreError: 81755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2725.125692] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.151452] Lustre: MGS: Client a627d25d-8b31-444f-85b7-ec804677de73 (at 0@lo) reconnecting [ 2725.197801] LustreError: 81766:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2725.344574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2725.571682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.683674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2729.837064] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2746.344716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2746.346573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.347591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2746.347597] Lustre: Skipped 2 previous similar messages [ 2746.396135] Lustre: Skipped 1 previous similar message [ 2750.571782] Lustre: server umount lustre-OST0000 complete [ 2756.577332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.585952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.596591] Lustre: Skipped 1 previous similar message [ 2756.600255] Lustre: Skipped 1 previous similar message [ 2761.700982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.713125] Lustre: Skipped 2 previous similar messages [ 2762.545556] Lustre: server umount lustre-MDT0000 complete [ 2766.817547] LustreError: 83560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.839345] LustreError: 83560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2767.686336] Lustre: server umount lustre-MDT0001 complete [ 2771.463258] Lustre: server umount MGS complete [ 2782.169169] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:26:43 (1781227603) [ 2816.198907] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 2829.234472] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 2872.478969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.928939] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2889.351933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.693682] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2889.788250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2889.934650] Lustre: lustre-MDT0001: new disk, initializing [ 2890.032300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.078959] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2890.091728] Lustre: 88840:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2890.106811] LustreError: 88840:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2895.773328] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2907.787056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.266574] Lustre: lustre-OST0000: new disk, initializing [ 2908.274073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2908.282248] Lustre: 89588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2908.373660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.985449] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2916.897311] Lustre: 89911:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000066cd1b0f x1867752398692736/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781227756 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2924.978614] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2934.765800] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.808645] Lustre: MGS: Client 8c0a28d8-2df7-423b-adc5-807242933b15 (at 0@lo) reconnecting [ 2934.908686] Lustre: 90522:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2934.925953] LustreError: 88826:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2934.957451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2935.050410] Lustre: lustre-MDT0000: new disk, initializing [ 2935.272525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.312101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2935.348309] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2936.423050] Lustre: 90546:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001ec020da x1867752398699904/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781227776 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2940.876184] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 2941.606379] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2944.498427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2944.515050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2945.584917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2945.599468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2945.676741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2973.160424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.169357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.184024] Lustre: Skipped 1 previous similar message [ 2973.204924] Lustre: Skipped 1 previous similar message [ 2978.802171] Lustre: server umount lustre-OST0000 complete [ 2983.397136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.400647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2983.406454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.421901] Lustre: Skipped 3 previous similar messages [ 2989.351215] Lustre: server umount lustre-MDT0000 complete [ 2999.818370] Lustre: server umount lustre-MDT0001 complete [ 3004.035495] Lustre: server umount MGS complete [ 3016.087199] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:30:37 (1781227837) [ 3035.645162] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3039.062677] Key type lgssc unregistered [ 3039.420078] LNet: 93015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3039.424705] LNetError: 93015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3039.439551] LNet: Removed LNI 192.168.201.117@tcp [ 3040.923922] Key type .llcrypt unregistered [ 3040.941672] Key type ._llcrypt unregistered [ 3061.078553] Key type ._llcrypt registered [ 3061.082507] Key type .llcrypt registered [ 3061.186752] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3062.294129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3062.426258] alg: No test for adler32 (adler32-zlib) [ 3063.481725] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3063.728820] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3065.411553] Key type lgssc registered [ 3066.603110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3075.374882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3075.384337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.982577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.264832] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3087.334395] LustreError: 94223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.357514] LustreError: 94223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3092.471829] LustreError: 94222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.240523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.074830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.193255] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3108.584588] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3113.321586] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3121.747028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.211346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.033823] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3137.254267] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3146.825485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3156.968540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.974295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3156.988386] Lustre: Skipped 1 previous similar message [ 3156.991923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.504569] Lustre: server umount lustre-OST0000 complete [ 3167.203193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.209278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.214389] Lustre: Skipped 1 previous similar message [ 3167.234189] Lustre: Skipped 2 previous similar messages [ 3172.168897] Lustre: server umount lustre-MDT0000 complete [ 3172.323678] LustreError: 94222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.355363] LustreError: 94222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3176.032082] LustreError: 94203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227999 with bad export cookie 913292830292029555 [ 3176.038681] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.470055] Lustre: server umount lustre-MDT0001 complete [ 3188.164935] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3191.026653] Key type lgssc unregistered [ 3191.345562] LNet: 97421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3191.352157] LNetError: 97421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3191.405164] LNet: Removed LNI 192.168.201.117@tcp [ 3192.499259] Key type .llcrypt unregistered [ 3192.505266] Key type ._llcrypt unregistered [ 3209.994991] Key type ._llcrypt registered [ 3210.001490] Key type .llcrypt registered [ 3210.121877] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3211.411381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3211.585250] alg: No test for adler32 (adler32-zlib) [ 3212.631112] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3212.840902] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3214.503204] Key type lgssc registered [ 3216.529382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3222.974718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3222.986910] LustreError: 98486:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3222.987556] LustreError: 98487:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3223.010837] LustreError: 98486:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3223.021769] LustreError: 98486:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3223.022522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.040802] LustreError: 98486:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3224.589128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.532684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.887131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3237.601704] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3243.296436] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing lsmod [ 3246.600865] LustreError: 99652:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3246.662686] LustreError: 99659:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3246.670493] LustreError: 99652:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4945 [ 3246.679402] LustreError: 99652:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3246.695321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.706134] LustreError: 99652:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3247.216209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.194956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.200024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.221461] Lustre: Skipped 2 previous similar messages [ 3252.232984] Lustre: Skipped 2 previous similar messages [ 3256.535980] Lustre: server umount lustre-MDT0000 complete [ 3257.319280] LustreError: 98520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.356764] LustreError: 98520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3258.338743] LustreError: 99051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.357865] LustreError: 99051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3260.783584] LustreError: 98503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228084 with bad export cookie 16962575654232167907 [ 3260.786020] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.794115] LustreError: 98503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3262.497451] LustreError: 98955:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000000f045e3 x1867752969097088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3262.688914] Lustre: server umount lustre-MDT0001 complete [ 3277.643682] Lustre: server umount lustre-OST0000 complete [ 3287.802044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.307251] LustreError: 100725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.371852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.992249] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3293.666242] LustreError: 100726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.772927] LustreError: 100725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.479817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.085463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3306.279734] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3311.544782] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3315.297292] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3321.444917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3321.861023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.989916] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3333.169406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3335.462340] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3343.328738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.344683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.358074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.393573] Lustre: Skipped 1 previous similar message [ 3348.462957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3348.471683] Lustre: Skipped 2 previous similar messages [ 3348.860565] Lustre: server umount lustre-OST0000 complete [ 3353.573710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3353.585585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.593018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.640326] Lustre: Skipped 1 previous similar message [ 3358.700397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3358.703289] Lustre: Skipped 2 previous similar messages [ 3367.391150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3367.804811] Lustre: server umount lustre-MDT0000 complete [ 3368.933480] LustreError: 100725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.951404] LustreError: 100725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3372.049487] LustreError: 100707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228195 with bad export cookie 16962575654232168600 [ 3372.051568] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.061534] LustreError: 100707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.657886] Lustre: server umount lustre-MDT0001 complete [ 3385.907529] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3389.221883] Key type lgssc unregistered [ 3389.533660] LNet: 103905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3389.546598] LNetError: 103905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3389.571066] LNet: Removed LNI 192.168.201.117@tcp [ 3390.455847] Key type .llcrypt unregistered [ 3390.460860] Key type ._llcrypt unregistered [ 3416.951846] Key type ._llcrypt registered [ 3416.953337] Key type .llcrypt registered [ 3417.140968] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3418.763713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3418.800373] alg: No test for adler32 (adler32-zlib) [ 3419.993607] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3420.209824] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3421.895181] Key type lgssc registered [ 3423.337815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3432.022390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3432.039867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.691643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.635959] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3444.205455] LustreError: 105304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.228311] LustreError: 105304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3446.040785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.386265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3450.919615] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3456.667198] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3460.828312] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3468.002747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.406809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.517803] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3478.689048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3483.142831] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3491.248360] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3497.164631] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3504.609368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.614904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.619023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.644572] Lustre: Skipped 1 previous similar message [ 3509.072857] Lustre: server umount lustre-OST0000 complete [ 3514.852082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.905406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.918837] Lustre: Skipped 1 previous similar message [ 3519.969879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.989057] Lustre: Skipped 2 previous similar messages [ 3520.242477] Lustre: server umount lustre-MDT0000 complete [ 3524.415023] LustreError: 105285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228348 with bad export cookie 12497044147039045061 [ 3524.425775] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3525.047689] Lustre: server umount lustre-MDT0001 complete [ 3537.511303] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3540.471635] Key type lgssc unregistered [ 3540.797772] LNet: 108588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3540.816351] LNetError: 108588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3540.844523] LNet: Removed LNI 192.168.201.117@tcp [ 3541.782635] Key type .llcrypt unregistered [ 3541.784843] Key type ._llcrypt unregistered [ 3559.723765] Key type ._llcrypt registered [ 3559.725382] Key type .llcrypt registered [ 3559.880689] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3561.274655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.438583] alg: No test for adler32 (adler32-zlib) [ 3562.506878] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3562.771303] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3564.519161] Key type lgssc registered [ 3565.714861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3573.502550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3573.527066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.137314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.985287] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3584.490548] LustreError: 109790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.512639] LustreError: 109790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3589.489403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.603577] LustreError: 109791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.159724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.398855] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3602.618126] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3606.751102] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3614.329696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3614.716313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.847518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3621.549685] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3628.489710] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3640.292036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.302813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.321112] Lustre: Skipped 1 previous similar message [ 3644.588557] Lustre: server umount lustre-OST0000 complete [ 3650.535848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.573054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.575839] Lustre: Skipped 1 previous similar message [ 3655.648513] LustreError: 109786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.674838] LustreError: 109786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3655.743642] Lustre: server umount lustre-MDT0000 complete [ 3659.842949] LustreError: 109772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228483 with bad export cookie 18258284294785038783 [ 3659.859581] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3660.209577] Lustre: server umount lustre-MDT0001 complete [ 3672.516282] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3676.015404] Key type lgssc unregistered [ 3676.352579] LNet: 112977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3676.361626] LNetError: 112977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3676.390597] LNet: Removed LNI 192.168.201.117@tcp [ 3677.192064] Key type .llcrypt unregistered [ 3677.193826] Key type ._llcrypt unregistered [ 3702.265764] Key type ._llcrypt registered [ 3702.270150] Key type .llcrypt registered [ 3702.444711] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3703.768373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.827795] alg: No test for adler32 (adler32-zlib) [ 3705.102750] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3705.447171] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3707.127244] Key type lgssc registered [ 3708.517661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3717.807602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3717.830817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.269466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.279990] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3728.866404] LustreError: 114378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.907407] LustreError: 114378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3734.002688] LustreError: 114379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3739.114994] LustreError: 114378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3739.664348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.264934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3746.171489] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3752.260944] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3757.482439] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3765.062154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.490594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.734455] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3778.617271] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3789.069831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3796.465297] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3800.084361] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3800.095602] Lustre: Skipped 2 previous similar messages [ 3804.012952] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3804.017940] Lustre: Skipped 2 previous similar messages [ 3804.214110] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3804.258904] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3818.134344] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.17@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3818.144045] Lustre: Skipped 2 previous similar messages [ 3824.182753] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.17@tcp 192.168.0.[2,10]@tcp [ 3828.028696] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.17/24@tcp in log lustre-MDT0000 [ 3828.032947] Lustre: Skipped 2 previous similar messages [ 3834.441649] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.17/24@tcp [ 3834.450904] Lustre: Skipped 1 previous similar message [ 3840.480266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.483147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.505804] Lustre: Skipped 1 previous similar message [ 3845.610520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.624530] Lustre: Skipped 2 previous similar messages [ 3846.007697] Lustre: server umount lustre-OST0000 complete [ 3850.721217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.729503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.735402] Lustre: Skipped 1 previous similar message [ 3850.743980] Lustre: Skipped 1 previous similar message [ 3855.844854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3857.441505] Lustre: server umount lustre-MDT0000 complete [ 3860.961140] LustreError: 114379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.993960] LustreError: 114379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3862.707672] LustreError: 114360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228686 with bad export cookie 3743983725523466165 [ 3862.718299] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.693908] Lustre: server umount lustre-MDT0001 complete [ 3876.900254] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 3880.462414] Key type lgssc unregistered [ 3880.773185] LNet: 118908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3880.786871] LNetError: 118908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3880.807536] LNet: Removed LNI 192.168.201.117@tcp [ 3881.813703] Key type .llcrypt unregistered [ 3881.817156] Key type ._llcrypt unregistered [ 3913.144045] Key type ._llcrypt registered [ 3913.147415] Key type .llcrypt registered [ 3913.326117] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 3914.839288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3914.985223] alg: No test for adler32 (adler32-zlib) [ 3916.117214] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3916.369240] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 3918.111462] Key type lgssc registered [ 3919.307652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3928.455833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3928.486057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.051282] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3930.093484] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.17/24@tcp [ 3930.116782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.747832] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3940.332157] LustreError: 120454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.341138] LustreError: 120454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3943.947871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.170167] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3944.212657] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.17/24@tcp [ 3944.256833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3948.563093] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3956.165552] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3960.118798] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3966.267463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.537348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.421792] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 3974.715218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3977.342181] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3988.960661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.970317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.981121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.990136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.996992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.003767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3990.015602] Lustre: Skipped 1 previous similar message [ 3993.318563] Lustre: server umount lustre-OST0000 complete [ 4000.225038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.227949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.247157] Lustre: Skipped 1 previous similar message [ 4003.563586] Lustre: server umount lustre-MDT0000 complete [ 4005.344678] LustreError: 120451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.355869] LustreError: 120451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4006.970265] LustreError: 120436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228830 with bad export cookie 10458200339577650447 [ 4006.977777] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4006.984923] LustreError: 120436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4007.274188] Lustre: server umount lustre-MDT0001 complete [ 4016.569712] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4019.930431] Key type lgssc unregistered [ 4020.295182] LNet: 123685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4020.312790] LNetError: 123685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4020.337420] LNet: Removed LNI 192.168.201.117@tcp [ 4021.190362] Key type .llcrypt unregistered [ 4021.191990] Key type ._llcrypt unregistered [ 4046.532265] Key type ._llcrypt registered [ 4046.534672] Key type .llcrypt registered [ 4046.733881] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4064.068226] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4065.623570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4065.700845] alg: No test for adler32 (adler32-zlib) [ 4067.026821] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 4067.421855] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4069.207770] Key type lgssc registered [ 4070.650581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4107.565606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4107.601731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.905874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4108.957590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4109.040259] Lustre: lustre-MDT0000: new disk, initializing [ 4109.109211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.128906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4113.495328] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4124.840983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.982480] Lustre: 127069:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4125.054015] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4125.060230] Lustre: Skipped 1 previous similar message [ 4125.199498] Lustre: lustre-MDT0001: new disk, initializing [ 4125.368942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4125.428902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4125.461108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4131.278962] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4139.684803] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4143.279963] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4149.585682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4149.897624] Lustre: lustre-OST0000: new disk, initializing [ 4149.905354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4149.918711] Lustre: 128811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4150.065385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.478899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4155.495505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4155.593937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4157.158712] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4164.824879] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4170.093438] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4170.326308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4175.110653] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4175.362785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4181.472531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4181.479396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.483153] LustreError: Skipped 1 previous similar message [ 4181.493784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4181.504659] Lustre: Skipped 1 previous similar message [ 4183.333784] Lustre: server umount lustre-OST0000 complete [ 4191.713222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.720924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4191.732892] Lustre: Skipped 2 previous similar messages [ 4191.745647] Lustre: Skipped 1 previous similar message [ 4193.847564] Lustre: server umount lustre-MDT0000 complete [ 4196.832193] LustreError: 127080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.847197] LustreError: 127080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4197.580124] LustreError: 127059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229021 with bad export cookie 15766094229176628711 [ 4197.590208] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4197.594082] LustreError: 127059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4198.210195] Lustre: server umount lustre-MDT0001 complete [ 4207.220978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.756743] LustreError: 130881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.862624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.529343] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4213.222982] LustreError: 130882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.342581] LustreError: 130881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4223.463103] LustreError: 130882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.247884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.842652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4230.640867] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4238.007978] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4241.912854] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4249.466738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.591374] LustreError: 132476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.975549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.877348] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4263.668202] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4269.100862] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4269.367617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4274.147154] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4274.450498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4285.228482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4285.424684] Lustre: lustre-OST0001: new disk, initializing [ 4285.427838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4285.432677] Lustre: 133881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4285.509196] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.961422] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4292.687724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4292.713661] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4292.822175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4299.789492] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4300.143862] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4305.116193] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4305.532411] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4310.012442] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d1a09ff7000.ost_server_uuid 50 [ 4311.833338] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d1a09ff7000.ost_server_uuid in FULL state after 0 sec [ 4322.502989] Lustre: server umount lustre-OST0001 complete [ 4323.817868] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4323.821901] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.831645] LustreError: 132478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.933381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.951595] Lustre: Skipped 1 previous similar message [ 4328.965023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.973634] Lustre: Skipped 1 previous similar message [ 4332.864313] Lustre: server umount lustre-OST0000 complete [ 4337.637207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4337.646038] LustreError: Skipped 1 previous similar message [ 4337.649063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.660469] Lustre: Skipped 1 previous similar message [ 4337.669943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.919318] Lustre: server umount lustre-MDT0000 complete [ 4347.869165] LustreError: 130863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229171 with bad export cookie 15766094229176629593 [ 4347.880978] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.264861] Lustre: server umount lustre-MDT0001 complete [ 4370.276551] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4373.136924] Key type lgssc unregistered [ 4373.481696] LNet: 136719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4373.493735] LNetError: 136719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4374.565253] LNet: Removed LNI 192.168.201.117@tcp [ 4375.559177] Key type .llcrypt unregistered [ 4375.562700] Key type ._llcrypt unregistered [ 4421.755122] Key type ._llcrypt registered [ 4421.756851] Key type .llcrypt registered [ 4421.850252] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4437.233844] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4438.628320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4438.774041] alg: No test for adler32 (adler32-zlib) [ 4439.882116] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 4440.169903] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4441.887195] Key type lgssc registered [ 4443.277331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4478.730697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4478.748858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.051698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4480.087424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4480.177668] Lustre: lustre-MDT0000: new disk, initializing [ 4480.342427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.374727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4485.341954] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4496.436277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.560635] Lustre: 140843:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4496.621530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4496.624814] Lustre: Skipped 1 previous similar message [ 4496.680466] Lustre: lustre-MDT0001: new disk, initializing [ 4496.784877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4496.844799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4496.858197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4501.423877] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4509.836950] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4513.700971] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4520.653245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4520.894676] Lustre: lustre-OST0000: new disk, initializing [ 4520.898160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4520.905100] Lustre: 142584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4520.982652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.953321] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4527.279387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4527.296794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4527.418793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4534.456175] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4538.309052] Lustre: Failing over lustre-OST0000 [ 4538.507029] Lustre: server umount lustre-OST0000 complete [ 4556.760378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4557.054618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.103591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.130481] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4557.463256] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4558.111237] Lustre: 139067:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229364/real 1781229364] req@0000000014acdd6b x1867754255711872/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1781229380 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4559.138025] Lustre: 139065:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229366/real 1781229366] req@00000000abb0c912 x1867754255712128/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781229382 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4559.178471] Lustre: 139065:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4562.460347] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4562.464048] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4563.424472] Lustre: 139066:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229371/real 1781229371] req@00000000ba77ed3b x1867754255713664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781229387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4564.097306] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4568.510601] Lustre: Failing over lustre-MDT0000 [ 4568.544578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4568.548623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.585089] Lustre: Skipped 1 previous similar message [ 4568.596253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.817501] Lustre: server umount lustre-MDT0000 complete [ 4569.587658] Lustre: 139068:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229376/real 1781229376] req@000000001a9c1907 x1867754255714560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781229392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4569.638776] Lustre: 139068:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4572.643182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4572.657903] Lustre: Skipped 1 previous similar message [ 4577.762569] LustreError: 140850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.781515] LustreError: 140850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4582.882038] LustreError: 140855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.906751] LustreError: 140855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4588.005332] LustreError: 142040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.071136] LustreError: 142040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4589.023512] Lustre: 139065:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229396/real 1781229396] req@00000000ae032f70 x1867754255724800/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781229412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4589.049729] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4589.816669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4594.145241] LustreError: 140850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.186333] LustreError: 140850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4599.264343] LustreError: 140854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.270532] LustreError: 139064:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000475dd16a x1867754255730688/t0(0) o250->MGC192.168.201.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4599.278657] LustreError: 140854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4599.399321] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (not set up) [ 4599.762749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.843192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4600.406482] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4604.913474] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4604.930353] Lustre: Skipped 1 previous similar message [ 4604.965763] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4605.013595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4605.639269] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4615.136690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.142069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4615.151535] Lustre: Skipped 1 previous similar message [ 4615.165852] Lustre: Skipped 1 previous similar message [ 4620.221269] Lustre: server umount lustre-OST0000 complete [ 4625.376783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.377623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.402742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.413918] Lustre: Skipped 1 previous similar message [ 4630.024225] Lustre: server umount lustre-MDT0000 complete [ 4630.497135] LustreError: 145140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.497135] LustreError: 143391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.648028] LustreError: 140834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229457 with bad export cookie 11957346018290130459 [ 4633.651522] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4633.656174] LustreError: 140834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4633.967412] Lustre: server umount lustre-MDT0001 complete [ 4644.342462] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 4646.808388] Key type lgssc unregistered [ 4647.037526] LNet: 146078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4647.042580] LNetError: 146078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4647.058762] LNet: Removed LNI 192.168.201.117@tcp [ 4647.736757] Key type .llcrypt unregistered [ 4647.752342] Key type ._llcrypt unregistered [ 4684.017921] Key type ._llcrypt registered [ 4684.021822] Key type .llcrypt registered [ 4684.125091] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4685.972615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4686.009025] alg: No test for adler32 (adler32-zlib) [ 4687.146575] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 4687.404658] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 4689.159474] Key type lgssc registered [ 4690.923876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4699.500423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4699.550262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4701.059653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.785398] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4711.398502] LustreError: 147592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.422270] LustreError: 147592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4714.611673] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4716.518551] LustreError: 147593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.020601] LustreError: 147621:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4719.032305] LustreError: 147621:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4719.051458] LustreError: 147621:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4719.409337] Lustre: server umount lustre-MDT0000 complete [ 4735.259742] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 4746.002587] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 4781.534558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.804497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4781.827752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4781.905687] Lustre: lustre-MDT0000: new disk, initializing [ 4781.986797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.003709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4786.308226] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4797.514596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.683712] Lustre: 150777:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4797.733882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4797.737636] Lustre: Skipped 1 previous similar message [ 4797.844624] Lustre: lustre-MDT0001: new disk, initializing [ 4797.986657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4798.063053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4798.087872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4803.777623] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4811.222660] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4815.229261] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4823.353944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.625677] Lustre: lustre-OST0000: new disk, initializing [ 4823.629497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4823.638465] Lustre: 152517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4823.758804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.325759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4829.347417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4829.523136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4831.850317] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 4841.424682] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4845.569934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4847.675092] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:01:09 (1781229669) [ 5074.399670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5074.402452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.436141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5075.425748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5075.426747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5075.459912] Lustre: Skipped 1 previous similar message [ 5075.489658] Lustre: Skipped 2 previous similar messages [ 5077.958693] Lustre: server umount lustre-MDT0000 complete [ 5080.544613] LustreError: 150785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.572540] LustreError: 150785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5081.637253] LustreError: 150770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229905 with bad export cookie 3016867361679578493 [ 5081.637400] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5081.649359] LustreError: 150770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5085.663627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.664652] LustreError: 152527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5085.682938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5088.014160] Lustre: server umount lustre-MDT0001 complete [ 5098.409778] Lustre: server umount lustre-OST0000 complete [ 5107.552744] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5118.318172] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5158.263046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.727289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5158.783603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5158.969958] Lustre: lustre-MDT0000: new disk, initializing [ 5159.124984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.145105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5164.293967] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5176.734913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.844299] Lustre: 157038:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5176.894458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5176.898528] Lustre: Skipped 1 previous similar message [ 5177.002316] Lustre: lustre-MDT0001: new disk, initializing [ 5177.072954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5177.116228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5177.132561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5182.760747] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5191.364384] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5194.982650] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5202.033726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5202.306586] Lustre: lustre-OST0000: new disk, initializing [ 5202.316147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5202.325321] Lustre: 158783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5202.427713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.050885] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5208.673554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5208.679626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5208.813536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5215.236860] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5220.690126] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5220.984673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5226.468937] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5226.716064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5229.535933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5229.539311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.545489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.546524] Lustre: Skipped 1 previous similar message [ 5229.566673] Lustre: Skipped 2 previous similar messages [ 5234.656512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.664886] Lustre: Skipped 1 previous similar message [ 5242.847436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5243.115177] Lustre: server umount lustre-OST0000 complete [ 5248.994566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5249.006853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5249.030060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5249.034918] Lustre: Skipped 2 previous similar messages [ 5253.469810] Lustre: server umount lustre-MDT0000 complete [ 5255.136476] LustreError: 157047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.160644] LustreError: 157047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5258.066073] LustreError: 157032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230081 with bad export cookie 3016867361679737323 [ 5258.072435] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5258.076299] LustreError: 157032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5258.432660] Lustre: server umount lustre-MDT0001 complete [ 5268.897753] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:08:10 (1781230090) [ 5286.607473] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5297.697069] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5333.539824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.897523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5333.940562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5334.050773] Lustre: lustre-MDT0000: new disk, initializing [ 5334.218947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.259455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5339.646545] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5350.782891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.971361] Lustre: 163359:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5351.257043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5355.849278] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5363.046592] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5367.184564] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5373.691895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.966434] Lustre: lustre-OST0000: new disk, initializing [ 5373.968759] Lustre: Skipped 1 previous similar message [ 5373.971892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5373.974612] Lustre: Skipped 2 previous similar messages [ 5373.983359] Lustre: 165108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5382.740444] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5382.831577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5382.844961] Lustre: Skipped 1 previous similar message [ 5382.857555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5383.034916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5390.340971] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5398.498601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5398.498933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.511765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5398.514793] Lustre: Skipped 2 previous similar messages [ 5398.533754] Lustre: Skipped 3 previous similar messages [ 5403.594994] Lustre: server umount lustre-OST0000 complete [ 5414.181647] Lustre: server umount lustre-MDT0000 complete [ 5418.968462] LustreError: 163352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230242 with bad export cookie 3016867361679738205 [ 5418.976028] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.978774] LustreError: 163352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5419.333805] Lustre: server umount lustre-MDT0001 complete [ 5431.055792] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:10:52 (1781230252) [ 5446.269043] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 5458.017660] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5498.967673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.428239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5499.489910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5499.664591] Lustre: lustre-MDT0000: new disk, initializing [ 5499.810733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5499.817651] Lustre: Skipped 2 previous similar messages [ 5499.832116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5505.841670] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5518.720925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.891476] Lustre: 169285:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5519.230293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5523.973978] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5531.705841] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5535.646721] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5542.166189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.391903] Lustre: 171038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5548.999972] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5549.301951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5556.756103] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5564.898019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5564.905275] LustreError: Skipped 1 previous similar message [ 5564.910468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5564.915157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5564.915164] Lustre: Skipped 4 previous similar messages [ 5564.937113] Lustre: Skipped 3 previous similar messages [ 5570.768567] Lustre: server umount lustre-OST0000 complete [ 5580.258647] LustreError: 169298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.267127] LustreError: 169298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5580.390820] Lustre: server umount lustre-MDT0000 complete [ 5584.874439] LustreError: 171741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230408 with bad export cookie 3016867361679739290 [ 5584.880748] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.893791] LustreError: 171741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5585.227119] Lustre: server umount lustre-MDT0001 complete [ 5594.864324] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5597.230702] Key type lgssc unregistered [ 5597.447669] LNet: 172764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5597.453697] LNetError: 172764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5597.467880] LNet: Removed LNI 192.168.201.117@tcp [ 5598.350916] Key type .llcrypt unregistered [ 5598.352819] Key type ._llcrypt unregistered [ 5622.372146] Key type ._llcrypt registered [ 5622.375131] Key type .llcrypt registered [ 5622.499409] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5624.243181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5624.323984] alg: No test for adler32 (adler32-zlib) [ 5625.620490] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 5625.976488] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5627.656896] Key type lgssc registered [ 5629.256196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5637.913867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5637.937914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.261857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5643.929967] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5648.878777] LustreError: 174163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5648.900849] LustreError: 174163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5652.942742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.340485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5658.770839] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5665.560499] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5670.229968] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5677.575662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.062635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.241653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5684.926619] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5691.281107] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5713.889186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5713.903696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.930456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.528362] Lustre: server umount lustre-OST0000 complete [ 5720.052343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5720.056881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.066642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.100937] Lustre: Skipped 1 previous similar message [ 5725.153239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.164576] Lustre: Skipped 2 previous similar messages [ 5730.280745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.285961] Lustre: Skipped 1 previous similar message [ 5733.343448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5733.625281] Lustre: server umount lustre-MDT0000 complete [ 5735.397346] LustreError: 174875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.424643] LustreError: 174875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5737.607672] LustreError: 174146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230561 with bad export cookie 6375817122866405292 [ 5737.615576] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.616806] LustreError: 174146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5737.893941] Lustre: server umount lustre-MDT0001 complete [ 5748.700105] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 5752.307897] Key type lgssc unregistered [ 5752.558537] LNet: 177364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5752.571424] LNetError: 177364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5752.603810] LNet: Removed LNI 192.168.201.117@tcp [ 5753.745590] Key type .llcrypt unregistered [ 5753.748376] Key type ._llcrypt unregistered [ 5778.958399] Key type ._llcrypt registered [ 5778.960421] Key type .llcrypt registered [ 5779.079621] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 5780.397550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5780.596577] alg: No test for adler32 (adler32-zlib) [ 5781.617993] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 5781.878570] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 5783.543203] Key type lgssc registered [ 5784.796515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5793.186504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5793.200537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5794.534837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.835203] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5805.038774] LustreError: 178767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.078779] LustreError: 178767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5808.507531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.848352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5814.476538] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5820.057689] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5823.534965] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5831.235928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.746067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.550279] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5839.986888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5845.165177] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5853.335153] Lustre: lustre-OST0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 5854.690675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.700917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.717568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.719516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.721443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.746332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.753246] Lustre: Skipped 1 previous similar message [ 5858.248874] Lustre: server umount lustre-OST0000 complete [ 5864.168339] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d1a068cb000.ost_server_uuid 50 [ 5865.914966] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5880.800506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5880.808750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.823171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5887.015824] Lustre: server umount lustre-MDT0000 complete [ 5887.458701] LustreError: 178768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.490370] LustreError: 178768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5891.504627] LustreError: 178749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230715 with bad export cookie 4977063630322286234 [ 5891.505671] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5891.512786] LustreError: 178749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5891.954577] Lustre: server umount lustre-MDT0001 complete [ 5900.285702] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:18:42 (1781230722) [ 5906.309659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.672668] LustreError: 182243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.744468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.052927] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5912.032660] LustreError: 182244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.154945] LustreError: 182243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.444401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.809994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5924.946720] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5931.141357] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5935.250937] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5942.818582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5943.163754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.982308] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5952.381925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5955.190252] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5961.628779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5961.795642] Lustre: lustre-OST0001: new disk, initializing [ 5961.799526] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5961.804562] Lustre: 184748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5961.887163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5968.590975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5968.599397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5968.709411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5969.045340] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 5976.373429] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5979.111423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.118456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5979.120144] Lustre: Skipped 2 previous similar messages [ 5979.123047] Lustre: Skipped 4 previous similar messages [ 5985.247688] Lustre: server umount lustre-OST0000 complete [ 5987.421405] LustreError: 184299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.300216] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5991.472385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5995.486801] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5995.697108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6004.721689] LustreError: 183841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.743802] LustreError: 183841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6009.843139] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6009.851963] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.857871] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6009.863742] Lustre: Skipped 1 previous similar message [ 6009.881490] Lustre: Skipped 5 previous similar messages [ 6015.486571] Lustre: server umount lustre-OST0001 complete [ 6020.064468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.081565] Lustre: Skipped 1 previous similar message [ 6030.305806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6030.312583] Lustre: Skipped 7 previous similar messages [ 6033.375278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.669229] Lustre: server umount lustre-MDT0000 complete [ 6037.330170] LustreError: 182225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230860 with bad export cookie 4977063630322287305 [ 6037.344450] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.352846] LustreError: 182225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6037.830521] Lustre: server umount lustre-MDT0001 complete [ 6064.402867] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:21:25 (1781230885) [ 6072.102893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.276752] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6072.298712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6072.404190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.428526] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6087.131207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.289281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6087.340153] Lustre: 188042:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6093.412610] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6100.851798] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6106.879918] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6115.618652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.806935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6125.648513] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6126.195825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6132.390100] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6138.888949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.947421] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6139.214468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6139.217030] Lustre: Skipped 2 previous similar messages [ 6145.238675] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6146.239651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6150.658812] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6156.786019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.790700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.793586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.813794] Lustre: Skipped 1 previous similar message [ 6156.835182] LustreError: Skipped 1 previous similar message [ 6159.614750] Lustre: server umount lustre-OST0000 complete [ 6161.902486] LustreError: 189670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.935716] LustreError: 189670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6177.247812] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6177.261658] LustreError: Skipped 1 previous similar message [ 6182.257327] Lustre: server umount lustre-OST0001 complete [ 6192.724244] Lustre: server umount lustre-MDT0000 complete [ 6195.896727] LustreError: 188037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231019 with bad export cookie 4977063630322288712 [ 6195.906321] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6195.911304] LustreError: 188037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6196.262303] Lustre: server umount lustre-MDT0001 complete [ 6219.712869] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:24:01 (1781231041) [ 6235.818335] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 6244.714889] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 6281.252925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.503173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6281.541201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6281.644366] Lustre: lustre-MDT0000: new disk, initializing [ 6281.740753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.751900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6286.081620] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6297.614964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.759656] Lustre: 195666:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6297.830233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6297.833747] Lustre: Skipped 1 previous similar message [ 6297.936917] Lustre: lustre-MDT0001: new disk, initializing [ 6298.102597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6298.115603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6303.159819] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6311.919177] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6315.691206] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6322.948227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6323.430136] Lustre: lustre-OST0000: new disk, initializing [ 6323.440472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6323.447335] Lustre: 197408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6332.172457] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6333.537120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6333.558870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6333.787235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6341.967559] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6347.319426] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6347.572862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6351.904363] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6352.274281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6354.408040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.420164] Lustre: Skipped 4 previous similar messages [ 6354.429382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6354.433248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6354.447013] Lustre: Skipped 7 previous similar messages [ 6360.524526] Lustre: server umount lustre-OST0000 complete [ 6371.011809] Lustre: server umount lustre-MDT0000 complete [ 6374.379228] LustreError: 195659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231197 with bad export cookie 4977063630322290497 [ 6374.392469] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6374.397222] LustreError: 195659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6374.911464] Lustre: server umount lustre-MDT0001 complete [ 6383.131360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.482574] LustreError: 199478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.498347] LustreError: 199478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6388.026853] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6397.117836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6402.478469] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6409.233338] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6413.924787] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6421.712421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.378196] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6435.721851] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6440.529405] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6440.809350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6445.999382] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6446.295872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6449.124030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6449.136060] LustreError: Skipped 1 previous similar message [ 6449.145428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6449.157403] Lustre: Skipped 3 previous similar messages [ 6455.067675] Lustre: server umount lustre-OST0000 complete [ 6465.103623] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6472.408731] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6502.720028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.431183] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6515.387469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.411976] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6516.029814] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6520.824054] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6524.593600] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.624414] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6525.924882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6525.942493] LustreError: Skipped 1 previous similar message [ 6525.951497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6525.962276] Lustre: Skipped 9 previous similar messages [ 6531.333043] Lustre: server umount lustre-OST0000 complete [ 6541.624058] Lustre: server umount lustre-MDT0000 complete [ 6545.378972] LustreError: 199474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.406041] LustreError: 199474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6545.503648] LustreError: 199459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231369 with bad export cookie 4977063630322291379 [ 6545.508755] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.950631] Lustre: server umount lustre-MDT0001 complete [ 6555.945675] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:29:37 (1781231377) [ 6562.471878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.890551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.895528] Lustre: Skipped 6 previous similar messages [ 6566.989653] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6574.514320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.327341] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6584.553578] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6587.990815] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6594.823559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.094785] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6608.527033] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6613.159993] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6613.357799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6618.502667] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6618.779838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6627.285398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.465733] Lustre: lustre-OST0001: new disk, initializing [ 6627.469471] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6627.478337] Lustre: 207921:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6634.102862] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6635.144555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6635.154721] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6635.253834] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6641.580397] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6645.701493] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6645.889139] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6650.160231] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6650.436429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6655.974969] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.994522] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6656.009304] Lustre: Skipped 6 previous similar messages [ 6656.031423] LustreError: Skipped 1 previous similar message [ 6658.593130] Lustre: server umount lustre-OST0001 complete [ 6665.933502] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6666.233560] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6671.768852] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6671.997917] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6701.522217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.189027] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6707.214071] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6707.215133] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6707.239748] Lustre: Skipped 1 previous similar message [ 6709.039996] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6714.218833] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6715.706760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6723.621690] Lustre: server umount lustre-OST0001 complete [ 6747.389041] Lustre: server umount lustre-OST0000 complete [ 6757.062459] Lustre: server umount lustre-MDT0000 complete [ 6760.635619] LustreError: 210106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231584 with bad export cookie 4977063630322292338 [ 6760.647122] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.649663] LustreError: 210106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6767.070659] Lustre: server umount lustre-MDT0001 complete [ 6792.183690] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:33:33 (1781231613) [ 6798.608690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.020828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6799.072218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6803.996928] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6813.774660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.888795] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6813.920313] Lustre: 212875:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6819.889948] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6826.330978] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6830.550794] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6837.777813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6838.074629] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6844.596985] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6851.612546] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6860.864349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6860.940236] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6867.372635] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6873.520719] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6875.462582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6880.054296] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6880.274681] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6884.543955] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6884.798951] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6889.039822] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d1a06b72800.ost_server_uuid 50 [ 6890.793539] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d1a06b72800.ost_server_uuid in IDLE state after 0 sec [ 6892.053890] Lustre: Permanently deactivating lustre-OST0001 [ 6892.074598] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6897.134153] Lustre: Permanently reactivating lustre-OST0001 [ 6897.153927] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6897.158911] Lustre: Skipped 2 previous similar messages [ 6902.755126] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6902.761523] Lustre: Skipped 15 previous similar messages [ 6907.916671] Lustre: server umount lustre-OST0001 complete [ 6908.415214] LustreError: 214492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.440651] LustreError: 214492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 6913.517989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6913.525328] Lustre: Skipped 9 previous similar messages [ 6918.488836] Lustre: server umount lustre-OST0000 complete [ 6937.055416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6937.491967] Lustre: server umount lustre-MDT0000 complete [ 6942.289208] LustreError: 214511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231765 with bad export cookie 4977063630322293633 [ 6942.302291] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.708807] Lustre: server umount lustre-MDT0001 complete [ 6967.290557] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:36:29 (1781231789) [ 6975.780736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.075831] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6976.103129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6976.112540] Lustre: Skipped 2 previous similar messages [ 6980.509099] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6988.751291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.821796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6988.843720] Lustre: 218988:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6993.693096] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 6998.859891] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7003.004129] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7011.126451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.355187] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7011.376702] Lustre: Permanently deactivating lustre-OST0000 [ 7011.392610] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7017.907879] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7018.023494] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7024.969731] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7034.546262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.612776] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7041.319266] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7043.847782] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7047.783803] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7049.584473] Lustre: Permanently reactivating lustre-OST0000 [ 7049.619978] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7049.629128] Lustre: Skipped 2 previous similar messages [ 7052.546379] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7052.552244] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000bc6fba24, cur 1781231876 deadline 1781231871 last 1781231841 [ 7052.642968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7064.544361] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7064.554744] LustreError: Skipped 6 previous similar messages [ 7066.052624] Lustre: server umount lustre-OST0001 complete [ 7076.222321] Lustre: server umount lustre-OST0000 complete [ 7086.830428] Lustre: server umount lustre-MDT0000 complete [ 7090.599320] LustreError: 218982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231914 with bad export cookie 4977063630322295306 [ 7090.602879] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7090.616343] LustreError: 218982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7090.975647] Lustre: server umount lustre-MDT0001 complete [ 7103.441229] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7105.973429] Key type lgssc unregistered [ 7106.210212] LNet: 223403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7106.221988] LNetError: 223403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7107.239087] LNet: Removed LNI 192.168.201.117@tcp [ 7108.135950] Key type .llcrypt unregistered [ 7108.137292] Key type ._llcrypt unregistered [ 7133.607664] Key type ._llcrypt registered [ 7133.609818] Key type .llcrypt registered [ 7133.741035] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7135.770781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7135.824755] alg: No test for adler32 (adler32-zlib) [ 7136.989671] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 7137.318260] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7139.039257] Key type lgssc registered [ 7140.583738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7153.603527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7153.664284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.403040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.114992] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7164.906780] LustreError: 224853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.936913] LustreError: 224853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7170.025443] LustreError: 224854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.682880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.883173] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7170.912689] Lustre: 224843:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7170.933706] Lustre: Permanently deactivating lustre-MDT0001 [ 7170.957341] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7171.331834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7177.599310] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7177.847658] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7184.996470] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7191.438346] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7198.898045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.282864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.305060] LustreError: 226471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.330203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7205.357136] LustreError: 226471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7205.954590] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7210.467903] LustreError: 226469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.167433] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7215.633922] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7220.722591] LustreError: 226540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.751406] LustreError: 226540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7222.738355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.037213] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7228.437830] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7228.445847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7229.373922] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7235.817476] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7239.788633] Lustre: Permanently reactivating lustre-MDT0001 [ 7239.837651] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7239.850320] Lustre: Skipped 2 previous similar messages [ 7241.734455] Lustre: lustre-MDT0001: Client 128e8702-9587-47ed-85ce-ec88fd883916 (at 192.168.201.17@tcp) reconnecting [ 7247.878683] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7249.895831] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7250.125100] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.143485] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7250.161541] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7250.200119] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7257.738483] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7258.068709] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7263.020796] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7263.322764] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7266.214736] Lustre: Permanently deactivating lustre-MDT0001 [ 7266.234981] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7266.243189] Lustre: Skipped 2 previous similar messages [ 7272.159973] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7279.072500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7279.083688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.105577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.556510] Lustre: server umount lustre-MDT0000 complete [ 7280.613105] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.615656] LustreError: 224853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.630981] Lustre: Skipped 1 previous similar message [ 7280.654210] LustreError: 224853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7284.545441] LustreError: 225576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232108 with bad export cookie 15737980034587281131 [ 7284.561862] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7285.136926] Lustre: server umount lustre-MDT0001 complete [ 7291.576773] Lustre: server umount lustre-OST0000 complete [ 7305.647442] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:42:06 (1781232126) [ 7321.488413] Lustre: server umount lustre-OST0001 complete [ 7329.595457] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 7342.713416] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7384.457834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.699845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7384.706834] Lustre: Skipped 2 previous similar messages [ 7384.725263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7384.789820] Lustre: lustre-MDT0000: new disk, initializing [ 7384.855741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.868544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7389.709888] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7403.370511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.567846] Lustre: 232499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7403.618055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7403.624888] Lustre: Skipped 1 previous similar message [ 7403.730469] Lustre: lustre-MDT0001: new disk, initializing [ 7403.882095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7403.931085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7403.943597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7408.517492] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7416.177494] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7420.266819] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7427.199039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7427.447550] Lustre: lustre-OST0000: new disk, initializing [ 7427.451865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7427.458314] Lustre: 234242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7427.527728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.602959] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7434.343245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7434.355549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7434.484965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7442.172687] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7444.361753] LustreError: 232507:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7454.038045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.187720] Lustre: lustre-OST0001: new disk, initializing [ 7454.191217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7454.195683] Lustre: 235243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7454.383605] LustreError: 232507:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7454.421020] LustreError: 232508:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7460.539367] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7462.583245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7462.602469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7462.729524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7464.439191] LustreError: 232508:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7468.472504] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7470.279929] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7473.126245] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.137098] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7473.137222] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7473.143911] Lustre: Skipped 2 previous similar messages [ 7474.817507] Lustre: lustre-OST0001: Not available for connect from 192.168.201.17@tcp (stopping) [ 7474.825576] Lustre: Skipped 1 previous similar message [ 7478.242770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7478.256814] Lustre: Skipped 1 previous similar message [ 7478.857737] Lustre: server umount lustre-OST0001 complete [ 7479.903144] LustreError: 234227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.726716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7498.746727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.158661] Lustre: server umount lustre-OST0000 complete [ 7508.961526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.963688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7508.991087] Lustre: Skipped 2 previous similar messages [ 7509.005618] Lustre: Skipped 4 previous similar messages [ 7522.272334] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7522.511738] Lustre: server umount lustre-MDT0000 complete [ 7526.776452] LustreError: 232494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232350 with bad export cookie 15737980034587283945 [ 7526.777434] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.785886] LustreError: 232494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7541.215140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7541.527615] Lustre: server umount lustre-MDT0001 complete [ 7551.664591] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7554.540743] Key type lgssc unregistered [ 7554.748404] LNet: 237147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7554.757835] LNetError: 237147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7555.815694] LNet: Removed LNI 192.168.201.117@tcp [ 7556.807920] Key type .llcrypt unregistered [ 7556.811953] Key type ._llcrypt unregistered [ 7594.720919] Key type ._llcrypt registered [ 7594.722853] Key type .llcrypt registered [ 7594.835913] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7596.165461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7596.210921] alg: No test for adler32 (adler32-zlib) [ 7597.264465] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 7597.456510] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7599.087370] Key type lgssc registered [ 7600.238810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7608.038350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7608.063258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.410692] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7609.447314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7609.684400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.932927] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7622.969974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.060417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7623.083961] Lustre: 239339:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7623.271677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7627.356678] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7633.574593] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7639.272331] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7645.651506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7645.884721] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7646.089408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.568380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7653.955840] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7659.686549] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7714.280877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.281488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7714.292401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7719.343382] Lustre: server umount lustre-OST0000 complete [ 7724.206063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7736.078108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7736.353109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.096508] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7749.219653] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7754.743192] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7754.770986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7754.772469] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7790.562832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.572441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7790.575671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.575681] Lustre: Skipped 1 previous similar message [ 7790.590263] Lustre: Skipped 2 previous similar messages [ 7790.625030] LustreError: Skipped 2 previous similar messages [ 7794.092924] Lustre: server umount lustre-OST0000 complete [ 7800.800308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.802796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.814864] Lustre: Skipped 1 previous similar message [ 7800.833348] Lustre: Skipped 2 previous similar messages [ 7804.335137] Lustre: server umount lustre-MDT0000 complete [ 7808.406438] LustreError: 239332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232632 with bad export cookie 13081539376014665699 [ 7808.423616] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.757656] Lustre: server umount lustre-MDT0001 complete [ 7821.167901] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7824.218624] Key type lgssc unregistered [ 7824.686464] LNet: 244293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7824.705251] LNetError: 244293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7825.767787] LNet: Removed LNI 192.168.201.117@tcp [ 7826.791887] Key type .llcrypt unregistered [ 7826.793840] Key type ._llcrypt unregistered [ 7851.631810] Key type ._llcrypt registered [ 7851.638416] Key type .llcrypt registered [ 7851.742793] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7852.562868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7852.642228] alg: No test for adler32 (adler32-zlib) [ 7853.746718] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 7853.925283] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 7855.623157] Key type lgssc registered [ 7856.849355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7864.338699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7864.359983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.986601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.795394] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7875.559085] LustreError: 245697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.582975] LustreError: 245697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7880.152670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.437416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7884.701635] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7889.883781] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7893.732114] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7902.928652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7903.250330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.306697] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 7910.615841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7918.296558] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7946.727345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7946.730504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.753205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.765214] Lustre: Skipped 1 previous similar message [ 7946.781413] Lustre: Skipped 1 previous similar message [ 7949.079980] Lustre: server umount lustre-OST0000 complete [ 7956.964630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.981195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7956.990186] Lustre: Skipped 1 previous similar message [ 7959.609259] Lustre: server umount lustre-MDT0000 complete [ 7962.092085] LustreError: 245692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.124814] LustreError: 245692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7963.203628] LustreError: 245679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232786 with bad export cookie 13542278164606583840 [ 7963.212163] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7963.221664] LustreError: 245679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7963.812109] Lustre: server umount lustre-MDT0001 complete [ 7973.903861] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 7976.941818] Key type lgssc unregistered [ 7977.322537] LNet: 249646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7977.337807] LNetError: 249646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7977.364065] LNet: Removed LNI 192.168.201.117@tcp [ 7978.405105] Key type .llcrypt unregistered [ 7978.407151] Key type ._llcrypt unregistered [ 7996.098276] Key type ._llcrypt registered [ 7996.102358] Key type .llcrypt registered [ 7996.205030] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 7997.646866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7997.870725] alg: No test for adler32 (adler32-zlib) [ 7999.044565] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 7999.339947] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8000.999354] Key type lgssc registered [ 8002.213129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8010.422504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8010.449530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.928176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8016.217515] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8021.492438] LustreError: 250852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.513455] LustreError: 250852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8024.434391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8024.924713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8029.831778] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8035.621449] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8039.869957] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8048.506896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8049.013964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.208567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8057.431601] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8063.308265] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8075.746172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8075.754419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.770090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.249786] Lustre: server umount lustre-OST0000 complete [ 8081.377231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8081.380834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.388943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.914938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.917624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.033786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8088.045537] Lustre: Skipped 2 previous similar messages [ 8091.279369] Lustre: server umount lustre-MDT0000 complete [ 8093.161790] LustreError: 250846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.196917] LustreError: 250846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8096.278169] LustreError: 250833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232919 with bad export cookie 12625054091751919897 [ 8096.290671] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8096.298562] LustreError: 250833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8096.800562] Lustre: server umount lustre-MDT0001 complete [ 8110.098477] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 8113.789231] Key type lgssc unregistered [ 8114.384262] LNet: 254172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8114.389233] LNetError: 254172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8114.402513] LNet: Removed LNI 192.168.201.117@tcp [ 8115.683677] Key type .llcrypt unregistered [ 8115.685498] Key type ._llcrypt unregistered [ 8141.151588] Key type ._llcrypt registered [ 8141.153728] Key type .llcrypt registered [ 8141.254756] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8142.865879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8142.920103] alg: No test for adler32 (adler32-zlib) [ 8143.930458] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 8144.141078] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8145.815708] Key type lgssc registered [ 8147.063983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8153.983507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8154.021641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.525708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.261206] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8165.863247] LustreError: 255573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8165.884290] LustreError: 255573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8168.999212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.337090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8174.622488] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8180.484466] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8184.724445] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8192.235823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.748215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8200.216445] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8200.976697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8206.855069] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8234.989899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8234.998225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8235.029064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.030740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.031078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.031264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8239.340847] Lustre: server umount lustre-OST0000 complete [ 8246.242342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.263946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.295159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.298020] Lustre: Skipped 1 previous similar message [ 8250.473713] Lustre: server umount lustre-MDT0000 complete [ 8252.384951] LustreError: 255572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.430086] LustreError: 255572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8254.257917] LustreError: 255554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233077 with bad export cookie 7824889922150084255 [ 8254.263477] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.268828] LustreError: 255554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8254.637870] Lustre: server umount lustre-MDT0001 complete [ 8266.738831] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 8269.762090] Key type lgssc unregistered [ 8269.998781] LNet: 259523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8270.012872] LNetError: 259523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8270.061665] LNet: Removed LNI 192.168.201.117@tcp [ 8271.140919] Key type .llcrypt unregistered [ 8271.145337] Key type ._llcrypt unregistered [ 8289.444443] Key type ._llcrypt registered [ 8289.446317] Key type .llcrypt registered [ 8289.523619] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8290.528385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8290.708817] alg: No test for adler32 (adler32-zlib) [ 8291.986731] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 8292.306828] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8294.031191] Key type lgssc registered [ 8295.812947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.260156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.278771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.115254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.142815] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8316.394877] LustreError: 260741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.411577] LustreError: 260741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8321.200196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.489500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8326.571719] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8332.463595] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8335.579604] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8341.965442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.320458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.191544] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8351.556297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8355.296855] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8367.080826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.085536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.097811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.098792] LustreError: Skipped 1 previous similar message [ 8367.108643] Lustre: Skipped 1 previous similar message [ 8372.219677] LustreError: 262358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8372.234763] LustreError: 262358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8372.522544] Lustre: server umount lustre-OST0000 complete [ 8377.316094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.331336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8377.339831] Lustre: Skipped 2 previous similar messages [ 8377.354591] Lustre: Skipped 1 previous similar message [ 8382.437153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8382.443340] Lustre: Skipped 1 previous similar message [ 8383.235438] Lustre: server umount lustre-MDT0000 complete [ 8386.964513] LustreError: 260708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233210 with bad export cookie 8892092044157545721 [ 8386.985867] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.534972] Lustre: server umount lustre-MDT0001 complete [ 8398.877863] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 8401.488908] Key type lgssc unregistered [ 8401.751087] LNet: 264085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8401.760363] LNetError: 264085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8401.779958] LNet: Removed LNI 192.168.201.117@tcp [ 8402.638061] Key type .llcrypt unregistered [ 8402.641816] Key type ._llcrypt unregistered [ 8418.396231] Key type ._llcrypt registered [ 8418.398015] Key type .llcrypt registered [ 8418.520648] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8435.236325] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 8450.650209] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8451.989501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8452.026348] alg: No test for adler32 (adler32-zlib) [ 8453.105784] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 8453.310483] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8454.951189] Key type lgssc registered [ 8456.156917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8488.363067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8488.399450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.631680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8489.657744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8489.716432] Lustre: lustre-MDT0000: new disk, initializing [ 8489.787973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8489.807678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8493.997602] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8507.740830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.971827] Lustre: 267602:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8508.102026] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8508.114429] Lustre: Skipped 1 previous similar message [ 8508.260387] Lustre: lustre-MDT0001: new disk, initializing [ 8508.418523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8508.472054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8508.501755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8512.982772] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8521.250395] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8525.176983] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8534.168765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8534.582788] Lustre: lustre-OST0000: new disk, initializing [ 8534.590197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8534.606848] Lustre: 269346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8534.738260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8541.789290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8541.806223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8541.946632] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8541.999500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8549.002849] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8553.178413] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8553.327708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8557.477772] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8557.671810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8562.666191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8562.675888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.708832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8562.726607] Lustre: Skipped 1 previous similar message [ 8562.746969] Lustre: Skipped 1 previous similar message [ 8566.026137] Lustre: server umount lustre-OST0000 complete [ 8572.898391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8572.903221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8572.943673] Lustre: Skipped 1 previous similar message [ 8576.787051] Lustre: server umount lustre-MDT0000 complete [ 8581.244237] LustreError: 269344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233404 with bad export cookie 4914966475231388222 [ 8581.251137] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8581.255259] LustreError: 269344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8581.932895] Lustre: server umount lustre-MDT0001 complete [ 8593.067193] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:03:34 (1781233414) [ 8600.954542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.733974] LustreError: 271610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.751911] LustreError: 271610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8601.815176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.018132] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8607.205372] LustreError: 271611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8612.323059] LustreError: 271610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.676251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.077529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.571578] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8630.542774] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8635.712866] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8643.171939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8643.605903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.047779] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8657.194258] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8672.227752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8672.230237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.232245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.232253] Lustre: Skipped 1 previous similar message [ 8672.240414] LustreError: Skipped 2 previous similar messages [ 8672.293844] Lustre: Skipped 1 previous similar message [ 8676.730995] Lustre: server umount lustre-OST0000 complete [ 8682.464786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.469575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8682.478771] Lustre: Skipped 1 previous similar message [ 8682.491112] Lustre: Skipped 2 previous similar messages [ 8687.588833] LustreError: 271606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8687.604731] LustreError: 271606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8687.754227] Lustre: server umount lustre-MDT0000 complete [ 8691.671492] LustreError: 271591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233515 with bad export cookie 4914966475231389104 [ 8691.677291] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8691.692979] LustreError: 271591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8692.069968] Lustre: server umount lustre-MDT0001 complete [ 8703.930528] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 8707.324753] Key type lgssc unregistered [ 8707.596851] LNet: 274814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8707.600248] LNetError: 274814:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8707.620659] LNet: Removed LNI 192.168.201.117@tcp [ 8708.362086] Key type .llcrypt unregistered [ 8708.366096] Key type ._llcrypt unregistered [ 8739.397187] Key type ._llcrypt registered [ 8739.399598] Key type .llcrypt registered [ 8739.523705] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 8755.314914] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 8756.521449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8756.557081] alg: No test for adler32 (adler32-zlib) [ 8757.642568] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 8757.860634] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 8759.519428] Key type lgssc registered [ 8760.580446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8802.441342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8802.490913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8803.771763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8803.810973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8803.911325] Lustre: lustre-MDT0000: new disk, initializing [ 8804.019571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.043139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8808.020519] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8819.023776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.120300] Lustre: 278606:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8819.178148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8819.182990] Lustre: Skipped 1 previous similar message [ 8819.246115] Lustre: lustre-MDT0001: new disk, initializing [ 8819.324825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8819.373990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8819.386279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8823.326386] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8830.232233] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8833.971372] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8840.331358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8840.568862] Lustre: lustre-OST2710: new disk, initializing [ 8840.572273] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8840.578664] Lustre: 280346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8840.674866] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8845.931410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8845.952578] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8846.072807] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8847.008032] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8854.617245] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8861.823283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.999450] Lustre: lustre-OST03e8: new disk, initializing [ 8862.003305] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8862.007397] Lustre: 281294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8862.097923] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8867.894694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8867.905168] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8867.996758] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8868.514459] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 8875.429580] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8883.813674] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8883.987708] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8888.633493] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8888.911850] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8893.310820] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8893.470486] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8898.604948] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8898.839284] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8903.657633] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.664732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8903.667201] Lustre: Skipped 2 previous similar messages [ 8903.674627] Lustre: Skipped 3 previous similar messages [ 8905.936262] Lustre: server umount lustre-MDT0000 complete [ 8909.435158] LustreError: 278597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233733 with bad export cookie 7218179644315079527 [ 8909.440064] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8909.443765] LustreError: 278597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8909.678099] Lustre: server umount lustre-MDT0001 complete [ 8924.341873] Lustre: server umount lustre-OST2710 complete [ 8936.752602] Lustre: server umount lustre-OST03e8 complete [ 8961.270638] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 8972.952952] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9006.366869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.582806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9006.623743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9006.740097] Lustre: lustre-MDT0000: new disk, initializing [ 9006.827576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9006.851899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9011.706863] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9023.830430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.993627] Lustre: 286332:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9024.051523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9024.057948] Lustre: Skipped 1 previous similar message [ 9024.150938] Lustre: lustre-MDT0001: new disk, initializing [ 9024.237125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9024.302550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9024.336871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9028.371755] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9035.798696] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9039.870218] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9046.990557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9047.222696] Lustre: lustre-OST0000: new disk, initializing [ 9047.225977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9047.231339] Lustre: 288073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9047.373842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.325889] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9054.258429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9054.269493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9054.383461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9060.602689] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9065.361970] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9065.544920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9069.286338] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9069.442788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9075.170082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9075.174527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.179257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9075.202789] Lustre: Skipped 2 previous similar messages [ 9075.217364] Lustre: Skipped 1 previous similar message [ 9077.169495] Lustre: server umount lustre-OST0000 complete [ 9085.408437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.412389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.428597] Lustre: Skipped 1 previous similar message [ 9085.452036] Lustre: Skipped 1 previous similar message [ 9087.501408] Lustre: server umount lustre-MDT0000 complete [ 9090.528608] LustreError: 286342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.546690] LustreError: 286342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9091.073810] LustreError: 287117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233914 with bad export cookie 7218179644315081207 [ 9091.075842] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9091.079890] LustreError: 287117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9091.477166] Lustre: server umount lustre-MDT0001 complete [ 9103.238890] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:12:04 (1781233924) [ 9105.301348] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9107.861037] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:12:08 (1781233928) [ 9111.887555] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9144.792321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.082604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9145.105540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9145.280971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.229153] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9159.235734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.336898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9159.386742] Lustre: 291771:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9164.168636] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9185.980183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.133274] Lustre: Denying initial registration attempt from nid 192.168.201.117@tcp, specified as failover [ 9186.141292] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9186.148940] LustreError: 293192:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9186.155472] LustreError: 293192:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9186.160415] LustreError: 293192:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9186.302444] Lustre: server umount lustre-OST0000 complete [ 9186.304486] LustreError: 293192:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9195.490270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9195.515108] LustreError: Skipped 1 previous similar message [ 9195.517158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9195.542394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9199.414978] Lustre: server umount lustre-MDT0000 complete [ 9202.152704] LustreError: 293230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.624719] LustreError: 291763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234027 with bad export cookie 7218179644315082082 [ 9203.633336] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9203.636856] LustreError: 291763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9204.081332] Lustre: server umount lustre-MDT0001 complete [ 9217.031693] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 9220.374233] Key type lgssc unregistered [ 9220.805612] LNet: 294298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9220.824258] LNetError: 294298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9220.853975] LNet: Removed LNI 192.168.201.117@tcp [ 9222.052647] Key type .llcrypt unregistered [ 9222.054490] Key type ._llcrypt unregistered [ 9235.466668] Key type ._llcrypt registered [ 9235.469206] Key type .llcrypt registered [ 9235.573844] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9236.737715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9236.977229] alg: No test for adler32 (adler32-zlib) [ 9238.097511] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 9238.266974] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 9239.919544] Key type lgssc registered [ 9241.368372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9290.417623] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9299.331717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9299.344504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.779876] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9300.818389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9300.963683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.647963] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9315.005396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.126996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9315.165938] Lustre: 297028:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9315.526191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9320.339899] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9326.532904] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9330.519892] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9350.199964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9350.593446] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9350.765528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9356.926277] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9362.377139] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9369.571603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9369.580393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.582224] LustreError: Skipped 1 previous similar message [ 9369.586482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.599572] Lustre: Skipped 1 previous similar message [ 9369.627775] Lustre: Skipped 1 previous similar message [ 9370.923468] Lustre: server umount lustre-OST0000 complete [ 9377.248492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9377.266565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9377.285684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9379.809068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9379.809679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.144201] Lustre: server umount lustre-MDT0000 complete [ 9385.349523] LustreError: 297022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234208 with bad export cookie 13929934383022849521 [ 9385.354158] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9385.361971] LustreError: 297022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9386.191848] Lustre: server umount lustre-MDT0001 complete [ 9397.843539] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 9400.991991] Key type lgssc unregistered [ 9401.337317] LNet: 300247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9401.344779] LNetError: 300247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9401.375830] LNet: Removed LNI 192.168.201.117@tcp [ 9402.325328] Key type .llcrypt unregistered [ 9402.327294] Key type ._llcrypt unregistered [ 9427.776561] Key type ._llcrypt registered [ 9427.778619] Key type .llcrypt registered [ 9427.871468] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9429.035187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9429.201252] alg: No test for adler32 (adler32-zlib) [ 9430.306989] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 9430.527346] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 9432.255187] Key type lgssc registered [ 9433.487765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9441.166520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9441.181956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9442.634431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9446.690803] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9453.027593] LustreError: 301648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.046139] LustreError: 301648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9454.687534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.117833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9459.384867] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9464.500206] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9469.081597] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9478.389500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9478.733938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.695775] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9492.665891] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9504.737720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.747297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9504.759628] Lustre: Skipped 2 previous similar messages [ 9504.769870] Lustre: Skipped 2 previous similar messages [ 9505.930260] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.17@tcp (stopping) [ 9509.139101] Lustre: server umount lustre-MDT0000 complete [ 9509.860395] LustreError: 301648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9509.871235] LustreError: 301648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9510.999389] LustreError: 301645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9513.002176] LustreError: 301630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234336 with bad export cookie 85815837852793624 [ 9513.006474] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9513.012035] LustreError: 301630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9514.978095] LustreError: 301644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.978274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9515.034976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9519.009874] LustreError: 303529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.010635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9519.032625] LustreError: 303529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9519.047508] Lustre: Skipped 1 previous similar message [ 9519.715351] Lustre: server umount lustre-MDT0001 complete [ 9528.525848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9537.801066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.988629] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x130e10cc351970a [ 9558.013641] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [ 9558.462085] LustreError: 303254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9558.555408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.608296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9564.337040] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9572.852230] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9573.606942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.846615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9578.319467] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9578.991518] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9584.231381] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9585.635807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9589.706887] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9591.392803] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9599.460072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.467280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.508570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.707386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.710285] Lustre: Skipped 3 previous similar messages [ 9611.231229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9611.478617] Lustre: server umount lustre-OST0000 complete [ 9619.944517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.969311] Lustre: Skipped 2 previous similar messages [ 9619.989097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9620.002826] LustreError: Skipped 1 previous similar message [ 9621.864291] Lustre: server umount lustre-MDT0000 complete [ 9625.057280] LustreError: 304907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.082446] LustreError: 304907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9626.216690] LustreError: 301629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234449 with bad export cookie 85815837852800778 [ 9626.217990] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9626.223633] LustreError: 301629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9626.608537] Lustre: server umount lustre-MDT0001 complete [ 9640.822577] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 9644.058060] Key type lgssc unregistered [ 9644.481044] LNet: 307337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9644.498183] LNetError: 307337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9644.530921] LNet: Removed LNI 192.168.201.117@tcp [ 9645.318294] Key type .llcrypt unregistered [ 9645.322543] Key type ._llcrypt unregistered [ 9669.207650] Key type ._llcrypt registered [ 9669.209689] Key type .llcrypt registered [ 9669.311314] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9670.373152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9670.538086] alg: No test for adler32 (adler32-zlib) [ 9671.613538] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 9672.012570] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 9673.735190] Key type lgssc registered [ 9674.778128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9682.540960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9682.566294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.054354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.879781] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9694.198784] LustreError: 308738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.232248] LustreError: 308738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9698.704438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.202207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9703.649350] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9708.877395] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9713.672643] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9718.754284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.768079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9723.442613] Lustre: server umount lustre-MDT0000 complete [ 9723.873267] LustreError: 308732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9723.893827] LustreError: 308732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9726.878341] LustreError: 308718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234550 with bad export cookie 13587864955857910001 [ 9726.892683] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9726.893684] LustreError: 308718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9727.189911] Lustre: server umount lustre-MDT0001 complete [ 9742.631639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.947887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9742.994825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9743.224425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9747.400344] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9756.474309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.563314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9756.634323] Lustre: 311097:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9757.170886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.972142] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9767.774871] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9771.551860] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9781.104925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.295522] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9781.305308] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9781.311379] LustreError: 312756:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9781.316110] LustreError: 312756:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9781.320527] LustreError: 312756:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9781.370845] Lustre: server umount lustre-OST0000 complete [ 9781.373171] LustreError: 312756:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9790.578396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9790.766295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9790.945175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9796.956524] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9800.349763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9802.110379] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9809.230224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9809.327181] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9809.392880] Lustre: lustre-OST0001: new disk, initializing [ 9809.396854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9809.404621] Lustre: 314184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9809.517231] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9814.584385] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [ 9818.207100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9818.214984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9818.336804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9819.380857] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9823.717087] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9823.730589] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.745278] Lustre: Skipped 1 previous similar message [ 9823.757593] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9823.768561] Lustre: Skipped 2 previous similar messages [ 9827.837993] Lustre: server umount lustre-OST0001 complete [ 9828.838881] LustreError: 313222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9828.876360] LustreError: 313222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9833.960295] LustreError: 314192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9833.963956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9833.965046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9833.981458] LustreError: 314192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9833.998696] Lustre: Skipped 2 previous similar messages [ 9837.850626] Lustre: server umount lustre-OST0000 complete [ 9844.193675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9844.197493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.213423] LustreError: Skipped 1 previous similar message [ 9844.217463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9844.217470] Lustre: Skipped 1 previous similar message [ 9844.268432] Lustre: Skipped 1 previous similar message [ 9847.787153] Lustre: server umount lustre-MDT0000 complete [ 9849.318129] LustreError: 314874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9849.328370] LustreError: 314874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9852.105856] LustreError: 311091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234675 with bad export cookie 13587864955857910554 [ 9852.120894] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9852.137385] LustreError: 311091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9852.704516] Lustre: server umount lustre-MDT0001 complete [ 9864.113515] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [ 9866.975594] Key type lgssc unregistered [ 9867.203108] LNet: 316012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9867.206355] LNetError: 316012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9868.263803] LNet: Removed LNI 192.168.201.117@tcp [ 9868.708161] Key type .llcrypt unregistered [ 9868.709820] Key type ._llcrypt unregistered [ 9935.634206] Key type ._llcrypt registered [ 9935.637497] Key type .llcrypt registered [ 9935.721264] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [ 9947.887567] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [ 9948.921667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9948.949068] alg: No test for adler32 (adler32-zlib) [ 9949.996816] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 9950.178684] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [ 9951.815173] Key type lgssc registered [ 9952.990761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9986.126101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9986.158648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.429685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9987.463373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9987.555226] Lustre: lustre-MDT0000: new disk, initializing [ 9987.707606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9987.736363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9992.036568] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10003.270183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.383153] Lustre: 320892:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10003.432947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10003.436616] Lustre: Skipped 1 previous similar message [10003.522817] Lustre: lustre-MDT0001: new disk, initializing [10003.671942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10003.714406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10003.728136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10008.683511] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10017.381984] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10022.350601] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10029.413916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10029.634712] Lustre: lustre-OST0000: new disk, initializing [10029.637601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10029.643639] Lustre: 322637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10029.726155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10036.109390] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10038.874594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10038.889818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10038.982708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10045.221368] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10050.381426] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10050.638807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10055.999574] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10056.274529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10059.746870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10059.767062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10059.788827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10064.698989] Lustre: server umount lustre-OST0000 complete [10069.987925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.002039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.003957] Lustre: Skipped 2 previous similar messages [10070.030019] Lustre: Skipped 2 previous similar messages [10075.108209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10075.115476] Lustre: Skipped 1 previous similar message [10080.229871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10080.241333] Lustre: Skipped 1 previous similar message [10084.320181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10084.635904] Lustre: server umount lustre-MDT0000 complete [10089.090553] LustreError: 321680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234912 with bad export cookie 11888546968311922164 [10089.093048] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10089.102253] LustreError: 321680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10089.472864] Lustre: server umount lustre-MDT0001 complete [10100.707862] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:28:42 (1781234922) [10109.498355] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:28:51 (1781234931) [10118.890711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.501723] LustreError: 325245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.517495] LustreError: 325245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10119.560699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10124.378152] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10124.779791] LustreError: 325246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.895414] LustreError: 325245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.467307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.795389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10138.461505] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10144.688731] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10149.036594] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10156.031374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10156.497061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10163.132409] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10169.019567] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10177.202605] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10178.879900] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10180.632546] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10182.854590] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10184.844164] Lustre: Failing over lustre-MDT0000 [10185.091570] Lustre: server umount lustre-MDT0000 complete [10185.188772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.194197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10185.194972] LustreError: 325240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.202092] Lustre: Skipped 1 previous similar message [10185.250800] LustreError: Skipped 1 previous similar message [10190.312337] LustreError: 327394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.334316] LustreError: 327394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10200.416309] Lustre: 319116:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235008/real 1781235008] req@0000000054f6ebc2 x1867760033403264/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781235024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10200.449493] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10200.467075] LustreError: 325241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.487555] LustreError: 325241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10203.902417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.790928] LustreError: 319114:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096e77d44 x1867760033408512/t0(0) o250->MGC192.168.201.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10211.182711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10211.266887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10213.471419] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10215.478308] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10216.438325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10216.479521] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10216.536561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10223.020931] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10224.677803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10228.321882] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10230.868490] Lustre: Failing over lustre-MDT0000 [10231.266907] Lustre: server umount lustre-MDT0000 complete [10231.782564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.787588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10231.808742] Lustre: Skipped 1 previous similar message [10231.810235] LustreError: 325246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.868616] LustreError: 325246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10248.165061] Lustre: 319117:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235055/real 1781235055] req@00000000910ec783 x1867760033423104/t0(0) o400->MGC192.168.201.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781235071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10248.217110] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.054678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.405741] LustreError: 319114:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000064b5486a x1867760033429248/t0(0) o250->MGC192.168.201.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10258.900312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10258.979671] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10260.583516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10263.656549] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10264.048348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10264.058636] Lustre: Skipped 2 previous similar messages [10264.097222] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10264.169562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10271.736644] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10273.301407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10279.402539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10279.408368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10279.415204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10279.457148] Lustre: Skipped 3 previous similar messages [10284.924915] Lustre: server umount lustre-MDT0000 complete [10288.425488] LustreError: 326861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235112 with bad export cookie 11888546968311924873 [10288.429800] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10288.439632] LustreError: 326861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10288.739902] Lustre: server umount lustre-MDT0001 complete [10303.262672] Lustre: server umount lustre-OST0000 complete [10318.547107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.912399] LustreError: 331023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.935590] LustreError: 331023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [10323.338611] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10332.510043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10332.778657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10332.785329] Lustre: Skipped 1 previous similar message [10337.091508] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10342.616943] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10346.633984] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10352.697419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10357.732797] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10362.873518] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10364.282543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10374.633319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10374.650407] Lustre: Skipped 1 previous similar message [10374.650826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10374.656196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10374.663902] LustreError: Skipped 1 previous similar message [10374.674262] Lustre: Skipped 6 previous similar messages [10379.322280] Lustre: server umount lustre-OST0000 complete [10383.847506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10389.130273] Lustre: server umount lustre-MDT0000 complete [10389.984276] LustreError: 331023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.000029] LustreError: 331023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10392.361480] LustreError: 331004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235215 with bad export cookie 11888546968311925622 [10392.369853] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10392.374749] LustreError: 331004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10392.886729] Lustre: server umount lustre-MDT0001 complete [10403.166513] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [10405.962254] Key type lgssc unregistered [10406.195263] LNet: 334200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10406.203612] LNetError: 334200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10407.270388] LNet: Removed LNI 192.168.201.117@tcp [10407.984743] Key type .llcrypt unregistered [10407.987652] Key type ._llcrypt unregistered [10433.955952] Key type ._llcrypt registered [10433.957386] Key type .llcrypt registered [10434.028226] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10435.145337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10435.301500] alg: No test for adler32 (adler32-zlib) [10436.378688] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [10436.565780] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [10438.231253] Key type lgssc registered [10439.553872] Lustre: Echo OBD driver; http://www.lustre.org/ [10447.763215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10447.777844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.261659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10453.231241] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10458.594391] LustreError: 335701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10458.621207] LustreError: 335701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10461.613092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.004848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10466.670349] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10472.171987] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10475.878525] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10482.410748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.743479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10489.252695] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10494.083131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10495.017794] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10507.209514] Lustre: server umount lustre-MDT0000 complete [10508.278420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10508.298146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.349601] LustreError: 336412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.795347] LustreError: 335702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.796569] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.817330] LustreError: 335702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10509.846761] Lustre: Skipped 1 previous similar message [10510.918319] LustreError: 335683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235334 with bad export cookie 16769883433623386574 [10510.932300] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10510.944618] LustreError: 335683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10511.217772] Lustre: server umount lustre-MDT0001 complete [10524.789930] Lustre: server umount lustre-OST0000 complete [10539.343277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10539.792565] LustreError: 339071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.807736] LustreError: 339071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10539.863652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10544.067639] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10545.125156] LustreError: 339072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10554.361869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10554.940778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10560.618111] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10566.703612] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10570.944601] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10579.130560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10579.428574] LustreError: 340666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.450418] LustreError: 340666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10579.578950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10584.716453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10587.213519] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10592.814880] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10605.541987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.544363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10605.565459] Lustre: Skipped 1 previous similar message [10605.583795] Lustre: Skipped 2 previous similar messages [10610.656925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.669135] Lustre: Skipped 2 previous similar messages [10611.554424] Lustre: server umount lustre-MDT0000 complete [10615.661827] LustreError: 339053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235439 with bad export cookie 16769883433623387687 [10615.676630] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10615.776470] LustreError: 339067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.776973] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.802296] Lustre: Skipped 1 previous similar message [10615.811846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10616.073417] Lustre: server umount lustre-MDT0001 complete [10621.656249] Lustre: server umount lustre-OST0000 complete [10638.185436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.815932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10643.427986] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10649.064424] LustreError: 342646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.080984] LustreError: 342646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10652.822865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10657.811121] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10664.083794] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10668.010222] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10674.669796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10674.999766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10675.006120] Lustre: Skipped 1 previous similar message [10681.858887] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10684.215852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10688.664272] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10694.631444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10694.642505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10694.660386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10694.681794] Lustre: Skipped 1 previous similar message [10699.989857] Lustre: server umount lustre-OST0000 complete [10709.472597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10709.477034] LustreError: Skipped 1 previous similar message [10709.478792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10709.504112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10709.507500] Lustre: Skipped 1 previous similar message [10711.366673] Lustre: server umount lustre-MDT0000 complete [10715.104181] LustreError: 342641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.113442] LustreError: 342641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10715.632927] LustreError: 342626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235539 with bad export cookie 16769883433623388702 [10715.639488] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10715.645393] LustreError: 342626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10716.067873] Lustre: server umount lustre-MDT0001 complete [10726.685465] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [10729.491195] Key type lgssc unregistered [10729.816192] LNet: 345820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10729.842487] LNetError: 345820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10729.859906] LNet: Removed LNI 192.168.201.117@tcp [10730.683496] Key type .llcrypt unregistered [10730.685358] Key type ._llcrypt unregistered [10761.194049] Key type ._llcrypt registered [10761.203466] Key type .llcrypt registered [10761.440444] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10762.892969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10763.098075] alg: No test for adler32 (adler32-zlib) [10764.433883] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [10764.771598] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [10766.583333] Key type lgssc registered [10768.364016] Lustre: Echo OBD driver; http://www.lustre.org/ [10777.857268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10777.905636] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.911308] LustreError: 347253:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10777.927168] LustreError: 347253:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10777.932866] LustreError: 347253:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10777.937043] LustreError: 347253:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10777.943338] LustreError: 347253:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10785.036198] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10785.040597] LustreError: 347650:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10785.053238] LustreError: 347650:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10785.058869] LustreError: 347650:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10785.064700] LustreError: 347650:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10785.073216] LustreError: 347650:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10806.667586] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [10809.844309] Key type lgssc unregistered [10810.146855] LNet: 348637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10810.157595] LNetError: 348637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10810.182820] LNet: Removed LNI 192.168.201.117@tcp [10811.223490] Key type .llcrypt unregistered [10811.225620] Key type ._llcrypt unregistered [10831.091087] Key type ._llcrypt registered [10831.093283] Key type .llcrypt registered [10831.196159] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [10847.933603] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [10849.053909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10849.219084] alg: No test for adler32 (adler32-zlib) [10850.444271] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [10850.661080] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [10852.303262] Key type lgssc registered [10853.380737] Lustre: Echo OBD driver; http://www.lustre.org/ [10886.010348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10886.066024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.363291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10887.410482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10887.528528] Lustre: lustre-MDT0000: new disk, initializing [10887.662449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10887.681975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10891.844314] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10901.890181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10901.998511] Lustre: 351763:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10902.070555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10902.073902] Lustre: Skipped 1 previous similar message [10902.155507] Lustre: lustre-MDT0001: new disk, initializing [10902.239333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10902.270724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10902.282949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10907.236474] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10915.535875] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10919.441297] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10927.148074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.454900] Lustre: lustre-OST0000: new disk, initializing [10927.458837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10927.504327] Lustre: 353505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10927.711750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10935.463179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10935.485694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10935.686253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10936.544475] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [10945.118416] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10951.197050] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10951.446968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10957.869549] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10958.308861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10966.498823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10966.501898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.509242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10966.523399] Lustre: Skipped 1 previous similar message [10967.614988] Lustre: server umount lustre-OST0000 complete [10971.620668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.635664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.643515] Lustre: Skipped 1 previous similar message [10971.652545] Lustre: Skipped 1 previous similar message [10976.737350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10976.744342] Lustre: Skipped 2 previous similar messages [10977.566693] Lustre: server umount lustre-MDT0000 complete [10981.411251] LustreError: 351753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235805 with bad export cookie 638810029227780246 [10981.415027] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10981.428417] LustreError: 351753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10981.919699] Lustre: server umount lustre-MDT0001 complete [10992.141522] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10994.222635] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:43:35 (1781235815) [11002.718712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.392229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11008.482529] LustreError: 355818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.504699] LustreError: 355818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11008.784402] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11012.579025] LustreError: 355819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.699725] LustreError: 355818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11019.050166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.451487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11024.053774] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11030.442271] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11034.227584] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11043.619578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.195575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11053.005495] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11060.613348] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11067.590187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.703873] Lustre: lustre-OST0001: new disk, initializing [11067.708731] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11067.712913] Lustre: 358314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11067.775123] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11074.561870] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11074.618747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11074.636182] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11074.830861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11082.319493] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11089.026446] Lustre: lustre-OST0001: Not available for connect from 192.168.201.17@tcp (stopping) [11090.404090] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11090.409936] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.415505] LustreError: Skipped 2 previous similar messages [11090.432619] Lustre: Skipped 1 previous similar message [11092.242203] Lustre: server umount lustre-OST0001 complete [11094.108778] LustreError: 357411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.640922] LustreError: 357411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.668608] LustreError: 357411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11109.856320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11109.861509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.870641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11109.872884] Lustre: Skipped 2 previous similar messages [11110.887952] LustreError: 357411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11110.903700] LustreError: 357411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11114.649901] Lustre: server umount lustre-OST0000 complete [11121.120156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.126216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.138194] Lustre: Skipped 2 previous similar messages [11121.153306] Lustre: Skipped 3 previous similar messages [11124.858042] Lustre: server umount lustre-MDT0000 complete [11127.961357] LustreError: 358313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235951 with bad export cookie 638810029227781128 [11127.966944] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11127.970678] LustreError: 358313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11134.483206] Lustre: server umount lustre-MDT0001 complete [11144.937632] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [11147.571633] Key type lgssc unregistered [11147.813733] LNet: 360174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11147.825471] LNetError: 360174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11147.840869] LNet: Removed LNI 192.168.201.117@tcp [11148.635684] Key type .llcrypt unregistered [11148.637897] Key type ._llcrypt unregistered [11202.070430] Key type ._llcrypt registered [11202.074981] Key type .llcrypt registered [11202.187856] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [11203.659288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11203.694377] alg: No test for adler32 (adler32-zlib) [11204.778455] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11205.036503] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [11206.711785] Key type lgssc registered [11208.191706] Lustre: Echo OBD driver; http://www.lustre.org/ [11216.505537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11216.539989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11217.941830] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11217.993102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11218.268941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11223.068446] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11233.199662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11233.408087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11233.458919] Lustre: 362958:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11233.845640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11237.968773] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11243.705131] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11248.093938] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11251.695306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.702404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11251.712057] Lustre: Skipped 1 previous similar message [11251.729066] Lustre: Skipped 1 previous similar message [11256.403322] Lustre: server umount lustre-MDT0000 complete [11259.824022] LustreError: 362949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236083 with bad export cookie 13022816190036234832 [11259.851900] LustreError: 362949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11259.852879] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11260.088393] Lustre: server umount lustre-MDT0001 complete [11269.220557] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:48:11 (1781236091) [11274.968141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.347444] LustreError: 365241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11275.372686] LustreError: 365241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11275.405709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11279.628971] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11280.868087] LustreError: 365242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11285.984823] LustreError: 365241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11290.260866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.689546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11296.215191] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11301.304536] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11305.292578] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11312.533403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11312.758434] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11312.982929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11318.635541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11324.731963] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11333.178102] Lustre: Permanently deactivating lustre-OST0000 [11333.201974] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11341.927195] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11343.060696] LustreError: 367889:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11343.067384] LustreError: 367889:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11343.072370] LustreError: 367889:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11345.962894] LustreError: 367937:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11345.975296] LustreError: 367937:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11354.583811] Lustre: server umount lustre-OST0000 complete [11358.037277] LustreError: 368188:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11358.049774] LustreError: 368188:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11362.272610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11362.282846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.300172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11364.321094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11364.328374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.440706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11369.451633] Lustre: Skipped 2 previous similar messages [11370.729826] Lustre: server umount lustre-MDT0000 complete [11374.503216] LustreError: 365223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236198 with bad export cookie 13022816190036235413 [11374.511135] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11374.521078] LustreError: 365223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11384.949909] Lustre: server umount lustre-MDT0001 complete [11394.937182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.478914] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11402.255624] LustreError: 369330:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11409.598150] Lustre: 369669:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.117@tcp [11412.146913] Lustre: server umount MGS complete [11424.421070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11425.082622] LustreError: 370394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.118206] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11425.125502] Lustre: Skipped 1 previous similar message [11425.226978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11430.222298] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11430.374550] LustreError: 370395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.865432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11439.063979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11443.946028] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11450.444699] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11454.728315] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11456.653423] Lustre: Permanently reactivating lustre-OST0000 [11456.670959] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11456.674793] Lustre: Skipped 2 previous similar messages [11484.132092] LustreError: 371947:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11492.397812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.766025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11492.773322] Lustre: Skipped 1 previous similar message [11499.263254] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11505.533754] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11514.336738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11514.337363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11514.354855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11514.390408] Lustre: Skipped 1 previous similar message [11516.395146] Lustre: server umount lustre-OST0000 complete [11520.992237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11521.001344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.577595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11524.583883] Lustre: Skipped 3 previous similar messages [11526.988508] Lustre: server umount lustre-MDT0000 complete [11529.697051] LustreError: 372431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.710039] LustreError: 372431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11531.502909] LustreError: 370376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236355 with bad export cookie 13022816190036236463 [11531.517746] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11531.932611] Lustre: server umount lustre-MDT0001 complete [11544.839746] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [11547.407356] Key type lgssc unregistered [11547.696919] LNet: 374401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11547.700966] LNetError: 374401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11547.715331] LNet: Removed LNI 192.168.201.117@tcp [11548.395796] Key type .llcrypt unregistered [11548.397883] Key type ._llcrypt unregistered [11565.719444] Key type ._llcrypt registered [11565.720875] Key type .llcrypt registered [11565.864783] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [11580.814690] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [11582.098538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11582.121726] alg: No test for adler32 (adler32-zlib) [11583.400243] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11583.670505] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [11585.391171] Key type lgssc registered [11586.541608] Lustre: Echo OBD driver; http://www.lustre.org/ [11624.682042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11624.719268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.056781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11626.103514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11626.205571] Lustre: lustre-MDT0000: new disk, initializing [11626.300199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11626.318876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11630.845400] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11642.101823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11642.346386] Lustre: 377526:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11642.501680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11642.510612] Lustre: Skipped 1 previous similar message [11642.706876] Lustre: lustre-MDT0001: new disk, initializing [11642.907676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11642.987041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11643.000653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11647.952517] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11659.149447] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11663.587657] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11671.250385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.661964] Lustre: lustre-OST0000: new disk, initializing [11671.666365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11671.675420] Lustre: 379268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11671.830693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11677.256972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11677.270510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11677.846587] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11682.814384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11686.542446] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11691.402511] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11691.630825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11696.133761] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11696.369990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11698.149694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11698.150904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.151916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11698.156732] LustreError: Skipped 1 previous similar message [11698.171235] Lustre: Skipped 1 previous similar message [11703.275487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11703.294096] Lustre: Skipped 1 previous similar message [11704.402889] Lustre: server umount lustre-OST0000 complete [11708.387853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.399917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11708.402129] Lustre: Skipped 1 previous similar message [11708.425234] Lustre: Skipped 2 previous similar messages [11713.510173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11713.517272] Lustre: Skipped 1 previous similar message [11714.682917] Lustre: server umount lustre-MDT0000 complete [11719.458658] LustreError: 377518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236543 with bad export cookie 11172532510033139658 [11719.470229] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11719.476949] LustreError: 377518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11719.848409] Lustre: server umount lustre-MDT0001 complete [11732.111268] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11733.868580] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:55:55 (1781236555) [11741.509729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.917271] LustreError: 381582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.934766] LustreError: 381582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11742.000089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11745.772329] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11747.301597] LustreError: 381583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.416259] LustreError: 381582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.520410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11756.066414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11761.471925] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11768.317816] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11771.987103] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11777.815277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11778.139405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11783.926137] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11789.252617] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11791.054743] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11794.641205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11802.086636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11802.095560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11802.109150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11803.122410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11806.163750] Lustre: server umount lustre-OST0000 complete [11812.322625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11812.325163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11812.351034] Lustre: Skipped 1 previous similar message [11812.357511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11812.364829] Lustre: Skipped 2 previous similar messages [11816.739353] Lustre: server umount lustre-MDT0000 complete [11818.465127] LustreError: 382292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.479648] LustreError: 382292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11821.122860] LustreError: 381562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236644 with bad export cookie 11172532510033140540 [11821.129663] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11821.153899] LustreError: 381562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11821.526668] Lustre: server umount lustre-MDT0001 complete [11831.646903] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [11834.100727] Key type lgssc unregistered [11834.435584] LNet: 384949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11834.458019] LNetError: 384949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11835.493546] LNet: Removed LNI 192.168.201.117@tcp [11836.402587] Key type .llcrypt unregistered [11836.404644] Key type ._llcrypt unregistered [11866.750343] Key type ._llcrypt registered [11866.753193] Key type .llcrypt registered [11867.043940] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [11867.425533] Key type .llcrypt unregistered [11867.429112] Key type ._llcrypt unregistered [11885.141669] Key type ._llcrypt registered [11885.143705] Key type .llcrypt registered [11885.253397] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [11886.516958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11886.643146] alg: No test for adler32 (adler32-zlib) [11887.803544] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [11888.007278] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [11889.679161] Key type lgssc registered [11890.932147] Lustre: Echo OBD driver; http://www.lustre.org/ [11899.149579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11899.194702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.730135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11905.295629] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11911.151341] LustreError: 387128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11911.177981] LustreError: 387128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11916.238663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11916.257478] LustreError: 387129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11916.602273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11921.889704] LustreError: 387125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.915572] LustreError: 387125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11923.466721] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11925.985515] LustreError: 387128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.012814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11926.030460] LustreError: 387128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11929.342712] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11931.109175] LustreError: 387129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.133762] LustreError: 387129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11937.163400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.708041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11943.232531] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [11956.709086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11956.729214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.753408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11958.258133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11958.263273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11958.271093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11961.873370] Lustre: server umount lustre-OST0000 complete [11966.881772] Lustre: server umount lustre-MDT0000 complete [11968.482304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11968.483661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11968.525760] LustreError: 387128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.545822] LustreError: 387128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11971.203531] LustreError: 387108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236794 with bad export cookie 7273075555850545335 [11971.210208] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11971.216107] LustreError: 387108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11971.777496] Lustre: server umount lustre-MDT0001 complete [11985.162699] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [11988.216231] Key type lgssc unregistered [11988.532509] LNet: 390113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.536582] LNetError: 390113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.552703] LNet: Removed LNI 192.168.201.117@tcp [11989.352934] Key type .llcrypt unregistered [11989.354891] Key type ._llcrypt unregistered [12015.147123] Key type ._llcrypt registered [12015.149510] Key type .llcrypt registered [12015.286390] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12016.706499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12016.768015] alg: No test for adler32 (adler32-zlib) [12017.900794] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [12018.087143] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [12019.719612] Key type lgssc registered [12020.788728] Lustre: Echo OBD driver; http://www.lustre.org/ [12028.912832] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12028.926393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12044.451238] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12049.567210] LustreError: 391488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.117@tcp: failed processing log, type 4: rc = -110 [12076.192041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12082.979883] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12089.681793] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12096.668072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12097.403710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12097.524530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12102.625660] LustreError: 391513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.643856] LustreError: 391513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12102.752781] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12107.750247] LustreError: 391514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12110.821545] LustreError: 391513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12110.837944] LustreError: 391513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12111.973862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12112.635589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12118.078238] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12125.408258] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12129.243936] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12138.473565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12138.476514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.487438] LustreError: Skipped 1 previous similar message [12138.491304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12138.520413] Lustre: Skipped 1 previous similar message [12139.122729] Lustre: server umount lustre-OST0000 complete [12143.589051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.601818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12143.605553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12143.633571] Lustre: Skipped 2 previous similar messages [12145.822691] Lustre: server umount lustre-MDT0000 complete [12148.708949] LustreError: 392424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12148.728652] LustreError: 392424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12150.428022] LustreError: 391496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236974 with bad export cookie 1342818961568099159 [12150.454063] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12150.874225] Lustre: server umount lustre-MDT0001 complete [12163.803750] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [12166.494190] Key type lgssc unregistered [12166.899617] LNet: 394716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12166.902815] LNetError: 394716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12166.911698] LNet: Removed LNI 192.168.201.117@tcp [12167.449108] Key type .llcrypt unregistered [12167.450965] Key type ._llcrypt unregistered [12190.540346] Key type ._llcrypt registered [12190.543155] Key type .llcrypt registered [12190.662585] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12191.686705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12191.713934] alg: No test for adler32 (adler32-zlib) [12192.835224] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [12193.114984] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [12194.815230] Key type lgssc registered [12196.081795] Lustre: Echo OBD driver; http://www.lustre.org/ [12203.914201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12203.925496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12205.419652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12210.136657] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12214.756272] LustreError: 396121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.772384] LustreError: 396121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12219.010303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.533078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12224.047261] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12229.391972] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12233.784535] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12239.756308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12240.176286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12246.215298] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12249.476809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12252.088401] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12259.809862] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.812126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12259.827152] Lustre: Skipped 1 previous similar message [12259.849973] Lustre: Skipped 2 previous similar messages [12264.930303] LustreError: 396122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.966894] LustreError: 396122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12265.128090] Lustre: server umount lustre-MDT0000 complete [12267.399131] LustreError: 396102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.17@tcp arrived at 1781237091 with bad export cookie 1909357692829368877 [12269.727029] LustreError: 396104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237093 with bad export cookie 1909357692829367974 [12269.744600] LustreError: 396104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12269.747734] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12270.049045] LustreError: 396833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.064656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.072584] LustreError: 396833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12270.097610] Lustre: Skipped 1 previous similar message [12270.102720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12270.107039] Lustre: Skipped 1 previous similar message [12274.090909] LustreError: 396121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12274.091786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12274.122892] LustreError: 396121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12276.012662] Lustre: server umount lustre-OST0000 complete [12279.266087] LustreError: 396833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.281093] LustreError: 396833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12282.263441] Lustre: server umount lustre-MDT0001 complete [12293.509920] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [12296.767636] Key type lgssc unregistered [12297.083848] LNet: 399449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12297.090765] LNetError: 399449:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12297.120018] LNet: Removed LNI 192.168.201.117@tcp [12297.977815] Key type .llcrypt unregistered [12297.979383] Key type ._llcrypt unregistered [12322.405660] Key type ._llcrypt registered [12322.408752] Key type .llcrypt registered [12322.557675] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12323.918302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12324.106200] alg: No test for adler32 (adler32-zlib) [12325.185652] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [12325.514198] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [12327.223173] Key type lgssc registered [12328.573694] Lustre: Echo OBD driver; http://www.lustre.org/ [12336.815719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12336.848756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.466135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12343.056526] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12347.875867] LustreError: 400850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12347.895610] LustreError: 400850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12352.746255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12352.993034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12353.254313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12358.162144] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12366.405533] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12371.505900] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12379.398041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12379.837251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12382.967636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12387.364071] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12393.512638] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12399.584640] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12399.602043] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12399.637941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12401.258903] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [12403.681859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12403.693127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.226042] Lustre: server umount lustre-MDT0001 complete [12406.360919] LustreError: 400846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12408.805996] LustreError: 400846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12408.827701] LustreError: 400846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12413.925533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12413.935053] LustreError: 400849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12413.940594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12413.942418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12413.942425] Lustre: Skipped 1 previous similar message [12419.042601] LustreError: 400850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12419.721288] Lustre: server umount lustre-OST0000 complete [12423.670952] Lustre: server umount lustre-MDT0000 complete [12440.203993] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [12442.978686] Key type lgssc unregistered [12443.316571] LNet: 404180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12443.323448] LNetError: 404180:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12443.341377] LNet: Removed LNI 192.168.201.117@tcp [12443.976247] Key type .llcrypt unregistered [12443.978215] Key type ._llcrypt unregistered [12471.069229] Key type ._llcrypt registered [12471.070988] Key type .llcrypt registered [12471.207118] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [12486.401716] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12487.471972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12487.596890] alg: No test for adler32 (adler32-zlib) [12488.765936] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [12489.075666] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [12490.863302] Key type lgssc registered [12492.414112] Lustre: Echo OBD driver; http://www.lustre.org/ [12540.308861] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12555.622728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12555.666530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12556.982392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12557.027911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12557.118465] Lustre: lustre-MDT0000: new disk, initializing [12557.187197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12557.202638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12561.572160] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12576.579332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12576.887073] Lustre: lustre-OST0000: new disk, initializing [12576.891037] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12576.895238] Lustre: Skipped 1 previous similar message [12576.902304] Lustre: 409148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12576.981406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12582.954703] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12584.497693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12584.503754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12584.551053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12590.048370] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12604.598608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.670710] Lustre: 408147:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12604.720542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12604.771856] Lustre: lustre-MDT0001: new disk, initializing [12604.878419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12604.925495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12604.943858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12609.563849] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12618.752996] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12619.019630] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12622.826163] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12623.034198] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12628.960498] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12628.975168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12628.989654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12629.003150] Lustre: Skipped 1 previous similar message [12629.017646] Lustre: Skipped 1 previous similar message [12629.617561] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.17@tcp (stopping) [12633.892609] Lustre: server umount lustre-MDT0001 complete [12639.200593] LustreError: 408155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.222740] LustreError: 408155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12644.327553] LustreError: 408160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.347846] LustreError: 408160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12649.443759] LustreError: 408155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.452174] LustreError: 408155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12654.562562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.563162] LustreError: 408159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.583159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12659.683175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12659.695849] LustreError: 408160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.803785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12666.335694] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12666.438694] Lustre: server umount lustre-OST0000 complete [12669.925970] LustreError: 408159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.957834] LustreError: 408159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12676.709296] Lustre: server umount lustre-MDT0000 complete [12690.279468] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [12692.922819] Key type lgssc unregistered [12693.213851] LNet: 412715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12693.220707] LNetError: 412715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12693.264216] LNet: Removed LNI 192.168.201.117@tcp [12694.015939] Key type .llcrypt unregistered [12694.017701] Key type ._llcrypt unregistered [12735.808656] Key type ._llcrypt registered [12735.813709] Key type .llcrypt registered [12735.958824] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [12736.342306] Key type .llcrypt unregistered [12736.345536] Key type ._llcrypt unregistered [12754.216027] Key type ._llcrypt registered [12754.218774] Key type .llcrypt registered [12754.312510] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [12755.293067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12755.400688] alg: No test for adler32 (adler32-zlib) [12756.701378] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [12756.988489] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [12758.719393] Key type lgssc registered [12760.500937] Lustre: Echo OBD driver; http://www.lustre.org/ [12788.219323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12788.240523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12789.629956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12789.672732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12789.753678] Lustre: lustre-MDT0000: new disk, initializing [12789.901779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12789.931428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12793.505690] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12803.022194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12803.095337] Lustre: 416059:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12803.129148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12803.132298] Lustre: Skipped 1 previous similar message [12803.181576] Lustre: lustre-MDT0001: new disk, initializing [12803.244442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12803.281049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12803.289565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12806.864281] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12814.988778] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12819.287321] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12825.834524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12826.242230] Lustre: lustre-OST0000: new disk, initializing [12826.245430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12826.251718] Lustre: 417799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12826.316224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12832.383532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12832.388898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12832.503729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12832.566523] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12840.255567] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12848.097917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12848.103732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12848.126482] Lustre: Skipped 1 previous similar message [12848.141275] Lustre: Skipped 2 previous similar messages [12853.219992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12853.230284] Lustre: Skipped 2 previous similar messages [12854.021245] Lustre: server umount lustre-MDT0000 complete [12857.972124] LustreError: 416051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237681 with bad export cookie 14098872566953698031 [12857.976988] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12857.984973] LustreError: 416051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12858.265868] Lustre: server umount lustre-MDT0001 complete [12871.951444] Lustre: server umount lustre-OST0000 complete [12884.366090] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:15:04 (1781237704) [12893.735348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12894.778953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12899.572897] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12899.826666] LustreError: 419715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.852130] LustreError: 419715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12904.938722] LustreError: 419714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12908.216350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12908.615929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12913.263758] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12919.133659] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12923.236438] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12943.018644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12943.277641] Lustre: Found index 0 for lustre-OST0000, updating log [12943.459595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12949.699937] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [12955.426577] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12966.028615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12971.505217] Lustre: server umount lustre-OST0000 complete [12988.447497] Lustre: 414592:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781237795/real 1781237795] req@00000000d95fe471 x1867762976757248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781237811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12988.474608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12988.486416] Lustre: Skipped 1 previous similar message [12988.523719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12988.528611] Lustre: Skipped 1 previous similar message [12991.583147] Lustre: 414593:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781237799/real 1781237799] req@00000000b30d7b43 x1867762976757760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781237815 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12992.607166] Lustre: 414590:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781237800/real 1781237800] req@00000000c9f8a853 x1867762976758528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781237816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12994.016771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12997.791216] Lustre: 414590:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781237805/real 1781237805] req@0000000067c02ad0 x1867762976759680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781237821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12998.207134] Lustre: server umount lustre-MDT0000 complete [12999.160694] LustreError: 419709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12999.174995] LustreError: 419709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13001.890954] LustreError: 419696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237825 with bad export cookie 14098872566953699424 [13001.898342] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13001.902293] LustreError: 419696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13002.346745] Lustre: server umount lustre-MDT0001 complete [13007.904379] LNet: 1 local NIs in recovery (showing 1): 192.168.201.117@tcp [13012.013295] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:17:13 (1781237833) [13637.556915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13638.195027] LustreError: 425804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13638.226584] LustreError: 425804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13638.321214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13642.296920] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [13643.744738] LustreError: 425805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13647.866731] LustreError: 425804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13649.489823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13649.771241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13654.004659] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [13659.206113] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13662.376410] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13682.541662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13682.760610] Lustre: Found index 0 for lustre-OST0000, updating log [13682.997922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13686.066298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13688.936346] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [13694.224281] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13696.482940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13696.492750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13696.502952] Lustre: Skipped 2 previous similar messages [13696.508646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13696.513715] Lustre: Skipped 1 previous similar message [13697.056780] LNet: 1 local NIs in recovery (showing 1): 192.168.201.117@tcp [13702.459872] Lustre: server umount lustre-OST0000 complete [13708.196357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13708.198423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13708.205145] Lustre: Skipped 1 previous similar message [13708.218722] Lustre: Skipped 1 previous similar message [13709.224453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13711.333169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13711.350122] Lustre: Skipped 4 previous similar messages [13713.383364] LustreError: 425805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13713.408499] LustreError: 425805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13713.638881] Lustre: server umount lustre-MDT0000 complete [13717.383127] LustreError: 427439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238540 with bad export cookie 14098872566953700509 [13717.385761] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13717.392315] LustreError: 427439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13717.927183] Lustre: server umount lustre-MDT0001 complete [13748.805541] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [13760.590936] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [13799.847883] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:30:21 (1781238621) [13815.470962] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [13818.128672] Key type lgssc unregistered [13818.563824] LNet: 432023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13818.572893] LNetError: 432023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13944.645397] LNet: Removed LNI 192.168.201.117@tcp [13945.424230] Key type .llcrypt unregistered [13945.430842] Key type ._llcrypt unregistered [13961.642206] Key type ._llcrypt registered [13961.645407] Key type .llcrypt registered [13961.761276] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [13962.990930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13963.035449] alg: No test for adler32 (adler32-zlib) [13964.191231] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [13964.477391] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [13966.159820] Key type lgssc registered [13967.383723] Lustre: Echo OBD driver; http://www.lustre.org/ [13971.037852] LNet: Added LNI 192.168.201.117@tcp100 [8/256/0/180] [13971.862976] LNet: Added LNI 192.168.201.117@tcp101 [8/256/0/180] [13973.078493] LNet: Added LNI 192.168.201.117@tcp102 [8/256/0/180] [13974.095759] LNet: Added LNI 192.168.201.117@tcp103 [8/256/0/180] [13975.119743] LNet: Added LNI 192.168.201.117@tcp104 [8/256/0/180] [13976.018383] LNet: Added LNI 192.168.201.117@tcp105 [8/256/0/180] [13976.784923] LNet: Added LNI 192.168.201.117@tcp106 [8/256/0/180] [13977.699046] LNet: Added LNI 192.168.201.117@tcp107 [8/256/0/180] [13978.590893] LNet: Added LNI 192.168.201.117@tcp108 [8/256/0/180] [13979.626580] LNet: Added LNI 192.168.201.117@tcp109 [8/256/0/180] [13980.513672] LNet: Added LNI 192.168.201.117@tcp110 [8/256/0/180] [13981.436621] LNet: Added LNI 192.168.201.117@tcp111 [8/256/0/180] [13982.223382] LNet: Added LNI 192.168.201.117@tcp112 [8/256/0/180] [13983.028135] LNet: Added LNI 192.168.201.117@tcp113 [8/256/0/180] [13983.934270] LNet: Added LNI 192.168.201.117@tcp114 [8/256/0/180] [13984.783759] LNet: Added LNI 192.168.201.117@tcp115 [8/256/0/180] [13985.513731] LNet: Added LNI 192.168.201.117@tcp116 [8/256/0/180] [13986.439580] LNet: Added LNI 192.168.201.117@tcp117 [8/256/0/180] [13987.258218] LNet: Added LNI 192.168.201.117@tcp118 [8/256/0/180] [13988.080897] LNet: Added LNI 192.168.201.117@tcp119 [8/256/0/180] [13988.844679] LNet: Added LNI 192.168.201.117@tcp120 [8/256/0/180] [13989.727415] LNet: Added LNI 192.168.201.117@tcp121 [8/256/0/180] [13990.477383] LNet: Added LNI 192.168.201.117@tcp122 [8/256/0/180] [13991.434854] LNet: Added LNI 192.168.201.117@tcp123 [8/256/0/180] [13992.209745] LNet: Added LNI 192.168.201.117@tcp124 [8/256/0/180] [13993.150982] LNet: Added LNI 192.168.201.117@tcp125 [8/256/0/180] [13994.060878] LNet: Added LNI 192.168.201.117@tcp126 [8/256/0/180] [13995.001472] LNet: Added LNI 192.168.201.117@tcp127 [8/256/0/180] [13995.726130] LNet: Added LNI 192.168.201.117@tcp128 [8/256/0/180] [13996.660339] LNet: Added LNI 192.168.201.117@tcp129 [8/256/0/180] [13997.478467] LNet: Added LNI 192.168.201.117@tcp130 [8/256/0/180] [13998.403932] LNet: Added LNI 192.168.201.117@tcp131 [8/256/0/180] [13999.515813] LNet: Added LNI 192.168.201.117@tcp132 [8/256/0/180] [14000.831164] LNet: Added LNI 192.168.201.117@tcp133 [8/256/0/180] [14001.892952] LNet: Added LNI 192.168.201.117@tcp134 [8/256/0/180] [14002.879780] LNet: Added LNI 192.168.201.117@tcp135 [8/256/0/180] [14010.209453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14010.258588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14011.432172] Lustre: 435040:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.117@tcp: can't add NID 192.168.201.117@tcp130: rc = -75 [14011.545602] LNetError: 432497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14011.575878] LNetError: Protocol error connecting to 192.168.201.117@tcp123 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14011.595549] LNetError: Skipped 6 previous similar messages [14011.621166] LNet: 7 local NIs in recovery (showing 7): 192.168.201.117@tcp121, 192.168.201.117@tcp124, 192.168.201.117@tcp125, 192.168.201.117@tcp126, 192.168.201.117@tcp127, 192.168.201.117@tcp128, 192.168.201.117@tcp129 [14011.697870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14011.745980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14011.803867] Lustre: lustre-MDT0000: new disk, initializing [14011.888419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14011.907647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14012.640825] LNetError: 435071:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14012.675537] LNetError: 435071:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14012.701425] LNetError: Protocol error connecting to 192.168.201.117@tcp111 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14012.712106] LNetError: Skipped 4 previous similar messages [14016.625323] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14028.408841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14028.614163] Lustre: 435058:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14028.685657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14028.692373] Lustre: Skipped 1 previous similar message [14028.771878] Lustre: lustre-MDT0001: new disk, initializing [14028.803359] Lustre: 435841:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.117@tcp: can't add NID 192.168.201.117@tcp130: rc = -75 [14028.811455] Lustre: 435841:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14028.910509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14028.957242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14028.976889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14034.436722] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14036.594557] Lustre: 436187:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.117@tcp: can't add NID 192.168.201.117@tcp130: rc = -75 [14036.610743] Lustre: 436187:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14042.337973] LNetError: 432497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14042.360688] LNetError: 432497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14042.385512] LNetError: Protocol error connecting to 192.168.201.117@tcp111 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14042.599304] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14046.439861] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14048.485309] LNetError: 435068:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14048.505933] LNetError: 435068:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14048.517341] LNetError: Protocol error connecting to 192.168.201.117@tcp120 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14048.531994] LNetError: Skipped 1 previous similar message [14053.224686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14053.596482] Lustre: lustre-OST0000: new disk, initializing [14053.606658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14053.616292] Lustre: 436805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14053.759376] Lustre: 436810:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.117@tcp: can't add NID 192.168.201.117@tcp130: rc = -75 [14053.776115] Lustre: 436810:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14053.837187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14055.653697] LNetError: 435063:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14055.667429] LNetError: 435063:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14055.676962] LNetError: Protocol error connecting to 192.168.201.117@tcp111 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14055.691366] LNetError: Skipped 2 previous similar messages [14060.556357] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14063.730202] Lustre: 437247:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.117@tcp: can't add NID 192.168.201.117@tcp130: rc = -75 [14063.744034] Lustre: 437247:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14063.797062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14063.809097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14063.846125] LNetError: 432497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14063.856542] LNetError: 432497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [14063.884746] LNetError: Protocol error connecting to 192.168.201.117@tcp118 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14063.907132] LNetError: Skipped 3 previous similar messages [14064.073598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14069.131405] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14079.460807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14079.466420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14079.469226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14079.487439] Lustre: Skipped 1 previous similar message [14079.506385] Lustre: Skipped 1 previous similar message [14082.278419] LNetError: 432496:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14082.287266] LNetError: 432496:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14082.295577] LNetError: Protocol error connecting to 192.168.201.117@tcp127 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14082.304600] LNetError: Skipped 5 previous similar messages [14084.585594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14084.590569] Lustre: Skipped 1 previous similar message [14089.701947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14089.716762] Lustre: Skipped 1 previous similar message [14091.234928] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14091.550353] Lustre: server umount lustre-OST0000 complete [14099.937305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14099.940480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14099.950203] Lustre: Skipped 1 previous similar message [14099.974533] Lustre: Skipped 1 previous similar message [14101.989821] Lustre: server umount lustre-MDT0000 complete [14105.651081] LustreError: 435048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238929 with bad export cookie 14537187406570496277 [14105.664333] LustreError: 435048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14105.667582] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14106.170722] Lustre: server umount lustre-MDT0001 complete [14114.904311] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [14115.056459] LNetError: 435063:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.117 [14115.068320] LNetError: 435063:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14115.083143] LNetError: Protocol error connecting to 192.168.201.117@tcp118 at host 192.168.201.117:988: is it running a compatible version of Lustre? [14115.099053] LNetError: Skipped 7 previous similar messages [14117.635661] Key type lgssc unregistered [14118.002076] LNet: 438587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14118.015967] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14118.020490] LNet: Removed LNI 192.168.201.117@tcp [14119.072067] LNet: Removed LNI 192.168.201.117@tcp100 [14119.094863] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14119.104154] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14119.119784] LNet: Removed LNI 192.168.201.117@tcp101 [14120.159197] LNet: Removed LNI 192.168.201.117@tcp102 [14120.164642] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14120.170257] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14120.177147] LNet: Removed LNI 192.168.201.117@tcp103 [14121.183457] LNet: Removed LNI 192.168.201.117@tcp104 [14122.207274] LNet: Removed LNI 192.168.201.117@tcp105 [14122.213065] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14122.223881] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14122.239074] LNet: Removed LNI 192.168.201.117@tcp106 [14123.298319] LNet: Removed LNI 192.168.201.117@tcp107 [14124.319371] LNet: Removed LNI 192.168.201.117@tcp108 [14125.343351] LNet: Removed LNI 192.168.201.117@tcp109 [14126.367255] LNet: Removed LNI 192.168.201.117@tcp110 [14126.385621] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14126.393028] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14126.403894] LNet: Removed LNI 192.168.201.117@tcp111 [14127.457086] LNet: Removed LNI 192.168.201.117@tcp112 [14128.479608] LNet: Removed LNI 192.168.201.117@tcp113 [14129.568153] LNet: Removed LNI 192.168.201.117@tcp114 [14130.591194] LNet: Removed LNI 192.168.201.117@tcp115 [14131.615610] LNet: Removed LNI 192.168.201.117@tcp116 [14132.639363] LNet: Removed LNI 192.168.201.117@tcp117 [14133.663170] LNet: Removed LNI 192.168.201.117@tcp118 [14134.687740] LNet: Removed LNI 192.168.201.117@tcp119 [14134.693366] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14134.705317] LNetError: 438587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14134.719064] LNet: Removed LNI 192.168.201.117@tcp120 [14135.775195] LNet: Removed LNI 192.168.201.117@tcp121 [14136.799328] LNet: Removed LNI 192.168.201.117@tcp122 [14137.824749] LNet: Removed LNI 192.168.201.117@tcp123 [14138.848252] LNet: Removed LNI 192.168.201.117@tcp124 [14139.871772] LNet: Removed LNI 192.168.201.117@tcp125 [14140.895684] LNet: Removed LNI 192.168.201.117@tcp126 [14141.919904] LNet: Removed LNI 192.168.201.117@tcp127 [14142.944141] LNet: Removed LNI 192.168.201.117@tcp128 [14143.967243] LNet: Removed LNI 192.168.201.117@tcp129 [14143.971271] LNet: Removed LNI 192.168.201.117@tcp130 [14143.973970] LNet: Removed LNI 192.168.201.117@tcp131 [14143.979305] LNet: Removed LNI 192.168.201.117@tcp132 [14143.987592] LNet: Removed LNI 192.168.201.117@tcp133 [14143.993417] LNet: Removed LNI 192.168.201.117@tcp134 [14144.003314] LNet: Removed LNI 192.168.201.117@tcp135 [14144.879376] Key type .llcrypt unregistered [14144.884178] Key type ._llcrypt unregistered [14167.444193] Key type ._llcrypt registered [14167.445707] Key type .llcrypt registered [14167.517401] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [14181.929327] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [14182.966674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14182.982235] alg: No test for adler32 (adler32-zlib) [14184.066723] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [14184.280849] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [14185.999499] Key type lgssc registered [14187.259140] Lustre: Echo OBD driver; http://www.lustre.org/ [14223.118200] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:37:24 (1781239044) [14241.155428] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [14244.187230] Key type lgssc unregistered [14244.526976] LNet: 442625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14244.534896] LNetError: 442625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14244.549826] LNet: Removed LNI 192.168.201.117@tcp [14245.338923] Key type .llcrypt unregistered [14245.341666] Key type ._llcrypt unregistered [14260.765715] Key type ._llcrypt registered [14260.768391] Key type .llcrypt registered [14260.861873] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [14261.786965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14262.054826] alg: No test for adler32 (adler32-zlib) [14263.055782] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [14263.251964] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [14264.895245] Key type lgssc registered [14265.924981] Lustre: Echo OBD driver; http://www.lustre.org/ [14274.904793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14274.945186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14276.404454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14276.493898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14276.676171] Lustre: lustre-MDT0000: new disk, initializing [14276.863494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14276.902832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14281.423220] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14291.714343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14291.842572] Lustre: 443821:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14291.897046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14291.902465] Lustre: Skipped 1 previous similar message [14292.072434] Lustre: lustre-MDT0001: new disk, initializing [14292.200244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14292.279688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14292.290504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14296.795882] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14305.701825] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14309.851907] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14317.552169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14317.905468] Lustre: lustre-OST0000: new disk, initializing [14317.910676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14317.920700] Lustre: 445563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14318.032483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14323.790388] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14326.936208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14326.945243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14327.160411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14331.986660] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14336.001981] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing lnet_if_list [14340.936549] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14340.955802] LNet: Added LNI 192.168.201.117@tcp10 [8/256/0/180] [14345.843680] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14345.875260] LNet: Added LNI 192.168.201.117@tcp11 [8/256/0/180] [14350.207608] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14350.259591] LNet: Added LNI 192.168.201.117@tcp12 [8/256/0/180] [14355.170126] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14355.201471] LNet: Added LNI 192.168.201.117@tcp13 [8/256/0/180] [14360.140992] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14360.165505] LNet: Added LNI 192.168.201.117@tcp14 [8/256/0/180] [14364.269873] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14364.317262] LNet: Added LNI 192.168.201.117@tcp15 [8/256/0/180] [14369.509653] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14369.535713] LNet: Added LNI 192.168.201.117@tcp16 [8/256/0/180] [14374.259907] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14374.299411] LNet: Added LNI 192.168.201.117@tcp17 [8/256/0/180] [14378.570427] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14378.598488] LNet: Added LNI 192.168.201.117@tcp18 [8/256/0/180] [14382.898576] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14382.929550] LNet: Added LNI 192.168.201.117@tcp19 [8/256/0/180] [14387.784546] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14387.820124] LNet: Added LNI 192.168.201.117@tcp20 [8/256/0/180] [14393.182706] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14393.218104] LNet: Added LNI 192.168.201.117@tcp21 [8/256/0/180] [14398.285319] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14398.316240] LNet: Added LNI 192.168.201.117@tcp22 [8/256/0/180] [14403.229189] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14403.262934] LNet: Added LNI 192.168.201.117@tcp23 [8/256/0/180] [14408.564683] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14408.608900] LNet: Added LNI 192.168.201.117@tcp24 [8/256/0/180] [14414.175943] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14414.215855] LNet: Added LNI 192.168.201.117@tcp25 [8/256/0/180] [14419.554266] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14419.578440] LNet: Added LNI 192.168.201.117@tcp26 [8/256/0/180] [14424.823681] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14424.853568] LNet: Added LNI 192.168.201.117@tcp27 [8/256/0/180] [14431.180106] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14431.211064] LNet: Added LNI 192.168.201.117@tcp28 [8/256/0/180] [14437.048694] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14437.072377] LNet: Added LNI 192.168.201.117@tcp29 [8/256/0/180] [14444.510478] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14444.544307] LNet: Added LNI 192.168.201.117@tcp30 [8/256/0/180] [14450.001815] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14450.040660] LNet: Added LNI 192.168.201.117@tcp31 [8/256/0/180] [14456.751119] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14456.774779] LNet: Added LNI 192.168.201.117@tcp32 [8/256/0/180] [14464.144142] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14464.196449] LNet: Added LNI 192.168.201.117@tcp33 [8/256/0/180] [14470.960533] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14470.983304] LNet: Added LNI 192.168.201.117@tcp34 [8/256/0/180] [14477.486943] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14477.518346] LNet: Added LNI 192.168.201.117@tcp35 [8/256/0/180] [14484.059726] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14484.083071] LNet: Added LNI 192.168.201.117@tcp36 [8/256/0/180] [14490.487588] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14490.543728] LNet: Added LNI 192.168.201.117@tcp37 [8/256/0/180] [14497.981445] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14498.026580] LNet: Added LNI 192.168.201.117@tcp38 [8/256/0/180] [14505.140247] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14505.166693] LNet: Added LNI 192.168.201.117@tcp39 [8/256/0/180] [14511.801632] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14511.839637] LNet: Added LNI 192.168.201.117@tcp40 [8/256/0/180] [14519.806772] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14519.836467] LNet: Added LNI 192.168.201.117@tcp41 [8/256/0/180] [14527.880138] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14527.931778] LNet: Added LNI 192.168.201.117@tcp42 [8/256/0/180] [14535.047899] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14535.077545] LNet: Added LNI 192.168.201.117@tcp43 [8/256/0/180] [14543.104290] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14543.147617] LNet: Added LNI 192.168.201.117@tcp44 [8/256/0/180] [14550.056754] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14550.086873] LNet: Added LNI 192.168.201.117@tcp45 [8/256/0/180] [14557.128662] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14557.148381] LNet: Added LNI 192.168.201.117@tcp46 [8/256/0/180] [14564.418313] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14564.453032] LNet: Added LNI 192.168.201.117@tcp47 [8/256/0/180] [14572.662722] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14572.708346] LNet: Added LNI 192.168.201.117@tcp48 [8/256/0/180] [14584.134359] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14584.226882] LNet: Added LNI 192.168.201.117@tcp49 [8/256/0/180] [14594.225292] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14594.246043] LNet: Added LNI 192.168.201.117@tcp50 [8/256/0/180] [14615.008791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14615.019513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14615.033251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14619.105916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14619.109880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14619.141967] Lustre: Skipped 2 previous similar messages [14620.797264] Lustre: server umount lustre-MDT0000 complete [14624.845464] LustreError: 447389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239448 with bad export cookie 5453151417981066843 [14624.855266] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14624.866266] LustreError: 447389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14625.369245] Lustre: server umount lustre-MDT0001 complete [14642.673504] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [14644.707479] Lustre: 443095:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239452/real 1781239452] req@00000000e68ce6dc x1867764556008960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781239468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14644.738717] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14644.749765] Lustre: Skipped 1 previous similar message [14650.850901] Lustre: 443092:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239458/real 1781239458] req@0000000064bc6606 x1867764556009216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781239474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14651.745667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14656.031303] Lustre: 443092:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239463/real 1781239463] req@00000000fdc54a87 x1867764556009344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781239479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14661.087667] Lustre: 443094:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239468/real 1781239468] req@0000000062c3ec3f x1867764556009472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781239484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14670.369387] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4bad7c52972de8a3 [14670.383438] Lustre: MGC192.168.201.117@tcp: Connection restored to 0@lo (at 0@lo) [14670.718455] LustreError: 445556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14670.735106] LustreError: 445556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14670.784275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14673.891599] LustreError: 445557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14673.907256] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14675.835274] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14675.939161] LustreError: 445556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14681.056830] LustreError: 445557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14686.181146] LustreError: 445556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14687.051405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14687.541549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14689.574659] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14694.265594] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14703.010036] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14708.529722] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14733.800334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14733.817125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14733.855528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14739.964379] Lustre: server umount lustre-OST0000 complete [14746.087603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14746.106241] Lustre: Skipped 1 previous similar message [14746.112758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14746.121523] Lustre: Skipped 3 previous similar messages [14751.204942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14751.222186] Lustre: Skipped 1 previous similar message [14751.838497] Lustre: server umount lustre-MDT0000 complete [14756.207638] LustreError: 443814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239579 with bad export cookie 5453151417981069475 [14756.210387] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14756.214332] LustreError: 443814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14756.570780] Lustre: server umount lustre-MDT0001 complete [14769.549625] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [14773.040775] Key type lgssc unregistered [14773.610986] LNet: 471523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14773.629873] LNetError: 471523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14773.637648] LNet: Removed LNI 192.168.201.117@tcp [14773.645814] LNet: Removed LNI 192.168.201.117@tcp10 [14773.649751] LNet: Removed LNI 192.168.201.117@tcp11 [14773.654408] LNet: Removed LNI 192.168.201.117@tcp12 [14773.657606] LNet: Removed LNI 192.168.201.117@tcp13 [14773.664965] LNet: Removed LNI 192.168.201.117@tcp14 [14773.670087] LNet: Removed LNI 192.168.201.117@tcp15 [14773.680431] LNet: Removed LNI 192.168.201.117@tcp16 [14773.684728] LNet: Removed LNI 192.168.201.117@tcp17 [14773.690026] LNet: Removed LNI 192.168.201.117@tcp18 [14773.692039] LNet: Removed LNI 192.168.201.117@tcp19 [14773.707085] LNet: Removed LNI 192.168.201.117@tcp20 [14773.718908] LNet: Removed LNI 192.168.201.117@tcp21 [14773.724244] LNet: Removed LNI 192.168.201.117@tcp22 [14773.729336] LNet: Removed LNI 192.168.201.117@tcp23 [14773.731352] LNet: Removed LNI 192.168.201.117@tcp24 [14773.742027] LNet: Removed LNI 192.168.201.117@tcp25 [14773.750949] LNet: Removed LNI 192.168.201.117@tcp26 [14773.754516] LNet: Removed LNI 192.168.201.117@tcp27 [14773.757605] LNet: Removed LNI 192.168.201.117@tcp28 [14773.762105] LNet: Removed LNI 192.168.201.117@tcp29 [14773.765533] LNet: Removed LNI 192.168.201.117@tcp30 [14773.778199] LNet: Removed LNI 192.168.201.117@tcp31 [14773.780329] LNet: Removed LNI 192.168.201.117@tcp32 [14773.785471] LNet: Removed LNI 192.168.201.117@tcp33 [14773.788837] LNet: Removed LNI 192.168.201.117@tcp34 [14773.792263] LNet: Removed LNI 192.168.201.117@tcp35 [14773.794169] LNet: Removed LNI 192.168.201.117@tcp36 [14773.797863] LNet: Removed LNI 192.168.201.117@tcp37 [14773.800706] LNet: Removed LNI 192.168.201.117@tcp38 [14773.802968] LNet: Removed LNI 192.168.201.117@tcp39 [14773.809928] LNet: Removed LNI 192.168.201.117@tcp40 [14773.814154] LNet: Removed LNI 192.168.201.117@tcp41 [14773.816588] LNet: Removed LNI 192.168.201.117@tcp42 [14773.821174] LNet: Removed LNI 192.168.201.117@tcp43 [14773.825455] LNet: Removed LNI 192.168.201.117@tcp44 [14773.828494] LNet: Removed LNI 192.168.201.117@tcp45 [14773.831652] LNet: Removed LNI 192.168.201.117@tcp46 [14773.834584] LNet: Removed LNI 192.168.201.117@tcp47 [14773.838119] LNet: Removed LNI 192.168.201.117@tcp48 [14773.840747] LNet: Removed LNI 192.168.201.117@tcp49 [14773.849501] LNet: Removed LNI 192.168.201.117@tcp50 [14774.991179] Key type .llcrypt unregistered [14774.994612] Key type ._llcrypt unregistered [14797.315832] Key type ._llcrypt registered [14797.317606] Key type .llcrypt registered [14797.425342] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [14813.543951] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [14814.682603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14814.795921] alg: No test for adler32 (adler32-zlib) [14815.857379] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [14816.029612] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [14817.672412] Key type lgssc registered [14818.804731] Lustre: Echo OBD driver; http://www.lustre.org/ [14866.389098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14866.416760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14867.645193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14867.670852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14867.807873] Lustre: lustre-MDT0000: new disk, initializing [14867.938940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14867.984145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14873.160460] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14886.599553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14886.762816] Lustre: 474687:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14886.846134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14886.852270] Lustre: Skipped 1 previous similar message [14886.922566] Lustre: lustre-MDT0001: new disk, initializing [14887.042284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14887.097670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14887.129017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14892.791815] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14901.165836] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14905.590697] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14914.476156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14914.635803] Lustre: *** cfs_fail_loc=90f, val=0*** [14914.723895] Lustre: lustre-OST0000: new disk, initializing [14914.728989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14914.734296] Lustre: 476479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14914.875883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14921.223669] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [14924.893039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14924.903631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14925.038498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14929.406537] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14940.642877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14940.652413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14940.671901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14940.689268] Lustre: Skipped 1 previous similar message [14945.762493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14945.775507] Lustre: Skipped 1 previous similar message [14950.888077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14950.902848] Lustre: Skipped 1 previous similar message [14951.391239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14951.597840] Lustre: server umount lustre-OST0000 complete [14956.005613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14956.017717] Lustre: Skipped 1 previous similar message [14956.024213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14956.030188] Lustre: Skipped 1 previous similar message [14961.127597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14961.140035] Lustre: Skipped 1 previous similar message [14962.155740] Lustre: server umount lustre-MDT0000 complete [14966.359966] LustreError: 474679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239789 with bad export cookie 11099074868900050353 [14966.365379] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14966.371118] LustreError: 474679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14966.762932] Lustre: server umount lustre-MDT0001 complete [14977.654528] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [14980.492419] Key type lgssc unregistered [14980.822681] LNet: 478257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14980.828439] LNetError: 478257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14980.862467] LNet: Removed LNI 192.168.201.117@tcp [14981.913345] Key type .llcrypt unregistered [14981.916285] Key type ._llcrypt unregistered [15001.786817] Key type ._llcrypt registered [15001.789394] Key type .llcrypt registered [15001.874453] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [15003.086337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15003.110921] alg: No test for adler32 (adler32-zlib) [15004.200556] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [15004.454979] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [15006.175222] Key type lgssc registered [15007.271895] Lustre: Echo OBD driver; http://www.lustre.org/ [15015.427078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15015.443229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15016.897334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15021.003196] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15027.184782] LustreError: 479514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15027.198221] LustreError: 479514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15031.160432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15031.685214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15036.532613] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15042.183689] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15046.177371] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15054.372486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15054.854288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15062.640369] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15069.485525] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15082.577825] loop0: detected capacity change from 0 to 204800000 [15082.894527] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15082.902449] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15082.911836] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15090.837276] loop0: detected capacity change from 0 to 204800000 [15091.110705] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15091.130509] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15091.140903] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15098.956156] loop0: detected capacity change from 0 to 204800000 [15106.224347] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15106.335336] Lustre: fs15246-OST0000: new disk, initializing [15106.338697] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15106.345465] Lustre: 482767:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15106.411235] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15114.734827] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15125.752674] loop1: detected capacity change from 0 to 204800000 [15132.211793] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15132.379303] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15132.490162] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15132.590334] Lustre: fs15246-MDT0000: new disk, initializing [15132.837205] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15132.896452] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15133.077616] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15134.113071] Lustre: 483874:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d4201cdf x1867765333935488/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781239973 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15137.719290] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15139.182522] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15139.206540] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15139.273940] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15164.402735] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15164.437160] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15166.568438] Lustre: server umount fs15246-MDT0000 complete [15169.518197] LustreError: 484279:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15174.655666] LustreError: 481415:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15175.865780] Lustre: server umount fs15246-OST0000 complete [15189.984995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15189.990258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15190.011148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15192.750977] Lustre: server umount lustre-OST0000 complete [15200.224813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15200.243291] Lustre: Skipped 1 previous similar message [15200.245722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15200.259681] Lustre: Skipped 2 previous similar messages [15203.392633] Lustre: server umount lustre-MDT0000 complete [15205.346514] LustreError: 479514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15205.368170] LustreError: 479514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15207.844158] LustreError: 479495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240031 with bad export cookie 14727440746199143456 [15207.859670] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15207.865223] LustreError: 479495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15208.301193] Lustre: server umount lustre-MDT0001 complete [15220.756657] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [15224.134731] Key type lgssc unregistered [15224.525597] LNet: 486937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15224.531341] LNetError: 486937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15224.573237] LNet: Removed LNI 192.168.201.117@tcp [15225.556702] Key type .llcrypt unregistered [15225.560382] Key type ._llcrypt unregistered [15249.898500] Key type ._llcrypt registered [15249.900942] Key type .llcrypt registered [15249.998817] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [15251.184048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15251.261632] alg: No test for adler32 (adler32-zlib) [15252.304106] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [15252.582276] LNet: Added LNI 192.168.201.117@tcp [8/256/0/180] [15254.303206] Key type lgssc registered [15255.652508] Lustre: Echo OBD driver; http://www.lustre.org/ [15294.995038] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [15304.509134] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [15339.181917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15339.195378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15340.445340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15340.477482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15340.576331] Lustre: lustre-MDT0000: new disk, initializing [15340.659576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15340.675460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15344.503126] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15356.165269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15356.379540] Lustre: 491355:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15356.425579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15356.429161] Lustre: Skipped 1 previous similar message [15356.490404] Lustre: lustre-MDT0001: new disk, initializing [15356.567219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15356.617147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15356.639902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15362.326994] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15372.940317] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15377.480960] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15385.221107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15385.458674] Lustre: lustre-OST0000: new disk, initializing [15385.462708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15385.468965] Lustre: 493100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15385.599474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15392.901601] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_default_debug -1 all [15394.920492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15394.963426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15395.094953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15402.224872] Lustre: DEBUG MARKER: oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15406.959532] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15407.189385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15411.708614] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15411.878309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15415.792168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15415.798335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15415.803499] LustreError: Skipped 1 previous similar message [15415.808248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15415.838718] Lustre: Skipped 1 previous similar message [15420.128453] Lustre: server umount lustre-OST0000 complete [15426.024153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15426.040310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15426.058894] Lustre: Skipped 2 previous similar messages [15430.816927] Lustre: server umount lustre-MDT0000 complete [15435.398925] LustreError: 491348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240259 with bad export cookie 3480564997644138631 [15435.408569] LustreError: MGC192.168.201.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15435.409880] LustreError: 491348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15435.741849] Lustre: server umount lustre-MDT0001 complete [15462.895761] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing set_hostid [15474.418399] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing load_modules_local [15505.723621] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15181 sec ========= 00:58:47 (1781240327) [15507.117450] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:58:49 (1781240329) === [15510.415797] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:58:52 (1781240332) === [15545.401733] Lustre: DEBUG MARKER: oleg117-server.virtnet: executing unload_modules_local [15548.546819] Key type lgssc unregistered [15548.819673] LNet: 499542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15548.828515] LNetError: 499542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15548.845416] LNet: Removed LNI 192.168.201.117@tcp [15549.719579] Key type .llcrypt unregistered [15549.721273] Key type ._llcrypt unregistered