[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495762801 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003294] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008329] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012068] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014084] *** VALIDATE selinux *** [ 0.022839] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026810] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028093] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.033197] *** VALIDATE proc *** [ 0.034279] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042953] debug: unmapping init [mem 0xffffffff96659000-0xffffffff96660fff] [ 0.044177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045766] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055546] smp: Bringing up secondary CPUs ... [ 0.056684] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.061011] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.204024] node 0 deferred pages initialised in 137ms [ 0.209013] devtmpfs: initialized [ 0.210308] x86/mm: Memory block size: 128MB [ 0.213840] gcov: version magic: 0x41383552 [ 0.216312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.222278] pinctrl core: initialized pinctrl subsystem [ 0.223176] [ 0.223536] ************************************************************* [ 0.226016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228013] ** ** [ 0.230014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233014] ** ** [ 0.235008] ** This means that this kernel is built to expose internal ** [ 0.237014] ** IOMMU data structures, which may compromise security on ** [ 0.240012] ** your system. ** [ 0.242012] ** ** [ 0.244014] ** If you see this message and you are not debugging the ** [ 0.247013] ** kernel, report this immediately to your vendor! ** [ 0.249011] ** ** [ 0.252013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254008] ************************************************************* [ 0.256705] NET: Registered protocol family 16 [ 0.259466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.268028] cpuidle: using governor menu [ 0.269448] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272403] PCI: Using configuration type 1 for base access [ 0.274138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.284120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287082] cryptd: max_cpu_qlen set to 1000 [ 0.289218] ACPI: Added _OSI(Module Device) [ 0.290024] ACPI: Added _OSI(Processor Device) [ 0.292017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.298343] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304591] ACPI: Interpreter enabled [ 0.306063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308029] ACPI: Using IOAPIC for interrupt routing [ 0.309137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.312480] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323047] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.337493] acpiphp: Slot [2] registered [ 0.339101] acpiphp: Slot [5] registered [ 0.341167] acpiphp: Slot [6] registered [ 0.343137] acpiphp: Slot [7] registered [ 0.344197] acpiphp: Slot [8] registered [ 0.346146] acpiphp: Slot [9] registered [ 0.348168] acpiphp: Slot [10] registered [ 0.349137] acpiphp: Slot [3] registered [ 0.351117] acpiphp: Slot [4] registered [ 0.352121] acpiphp: Slot [11] registered [ 0.354128] acpiphp: Slot [12] registered [ 0.356189] acpiphp: Slot [13] registered [ 0.357146] acpiphp: Slot [14] registered [ 0.359125] acpiphp: Slot [15] registered [ 0.361113] acpiphp: Slot [16] registered [ 0.363149] acpiphp: Slot [17] registered [ 0.365139] acpiphp: Slot [18] registered [ 0.366181] acpiphp: Slot [19] registered [ 0.368095] acpiphp: Slot [20] registered [ 0.369136] acpiphp: Slot [21] registered [ 0.371195] acpiphp: Slot [22] registered [ 0.373108] acpiphp: Slot [23] registered [ 0.375137] acpiphp: Slot [24] registered [ 0.377149] acpiphp: Slot [25] registered [ 0.378198] acpiphp: Slot [26] registered [ 0.380110] acpiphp: Slot [27] registered [ 0.381091] acpiphp: Slot [28] registered [ 0.382087] acpiphp: Slot [29] registered [ 0.383146] acpiphp: Slot [30] registered [ 0.385096] acpiphp: Slot [31] registered [ 0.386070] PCI host bridge to bus 0000:00 [ 0.388021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.393021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.398030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.401033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.406074] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.409461] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.421875] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426033] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.431019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.435032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.438584] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.441877] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.445236] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.448908] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.455020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.469018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.475014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.479644] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.490017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.496016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.516018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525436] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.534015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.566019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.584714] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.593017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.605018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.627021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639076] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.647017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.653018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.673018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.680626] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.687017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.693017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.730249] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.738017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.745019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.767018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.779205] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.781367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784395] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.788408] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.790238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.795032] iommu: Default domain type: Passthrough [ 0.797545] SCSI subsystem initialized [ 0.800256] ACPI: bus type USB registered [ 0.801132] usbcore: registered new interface driver usbfs [ 0.803109] usbcore: registered new interface driver hub [ 0.805122] usbcore: registered new device driver usb [ 0.807187] pps_core: LinuxPPS API ver. 1 registered [ 0.809012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.812045] PTP clock support registered [ 0.815080] EDAC MC: Ver: 3.0.0 [ 0.817110] PCI: Using ACPI for IRQ routing [ 0.820071] NetLabel: Initializing [ 0.821000] NetLabel: domain hash size = 128 [ 0.822103] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.825088] NetLabel: unlabeled traffic allowed by default [ 0.828321] vgaarb: loaded [ 0.829454] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.830000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.840040] clocksource: Switched to clocksource kvm-clock [ 0.952389] VFS: Disk quotas dquot_6.6.0 [ 0.954141] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.956750] *** VALIDATE ramfs *** [ 0.958258] *** VALIDATE hugetlbfs *** [ 0.961672] pnp: PnP ACPI init [ 0.964327] pnp: PnP ACPI: found 6 devices [ 0.981664] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.986043] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.988564] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.990956] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.993798] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.996339] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.999539] NET: Registered protocol family 2 [ 1.002097] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.007798] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.011701] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.017113] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.020516] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023470] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027039] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029984] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.032915] NET: Registered protocol family 1 [ 1.035609] RPC: Registered named UNIX socket transport module. [ 1.037467] RPC: Registered udp transport module. [ 1.039207] RPC: Registered tcp transport module. [ 1.041226] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.043919] NET: Registered protocol family 44 [ 1.045563] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.047818] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.050115] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.052698] PCI: CLS 0 bytes, default 64 [ 1.055081] Unpacking initramfs... [ 3.499327] debug: unmapping init [mem 0xffff9e477cc54000-0xffff9e477ffbffff] [ 3.510134] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.515169] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.521475] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.660987] Initialise system trusted keyrings [ 5.664710] Key type blacklist registered [ 5.670795] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.708076] zbud: loaded [ 5.723914] *** VALIDATE nfs *** [ 5.727884] *** VALIDATE nfs4 *** [ 5.735261] pstore: using deflate compression [ 5.748330] Platform Keyring initialized [ 6.163223] NET: Registered protocol family 38 [ 6.166663] Key type asymmetric registered [ 6.169344] Asymmetric key parser 'x509' registered [ 6.172234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.182236] io scheduler mq-deadline registered [ 6.188627] io scheduler kyber registered [ 6.192447] io scheduler bfq registered [ 6.196904] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.206066] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.213678] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.220633] ACPI: Power Button [PWRF] [ 6.230571] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.245642] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.284610] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.317831] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.370983] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.408178] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.443341] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.456347] Non-volatile memory driver v1.3 [ 6.460209] Linux agpgart interface v0.103 [ 6.551682] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 6.559595] vda: detected capacity change from 0 to 76791808 [ 6.604640] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.611904] vdb: detected capacity change from 0 to 1073741824 [ 6.644879] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.650489] vdc: detected capacity change from 0 to 2621440000 [ 6.683555] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.688379] vdd: detected capacity change from 0 to 2621440000 [ 6.722808] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.731765] vde: detected capacity change from 0 to 4294967296 [ 6.761868] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.766341] vdf: detected capacity change from 0 to 4294967296 [ 6.779254] libphy: Fixed MDIO Bus: probed [ 6.790794] usbcore: registered new interface driver usbserial_generic [ 6.794867] usbserial: USB Serial support registered for generic [ 6.799601] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.806139] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.809103] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.812992] mousedev: PS/2 mouse device common for all mice [ 6.818925] rtc_cmos 00:05: RTC can wake from S4 [ 6.823766] rtc_cmos 00:05: registered as rtc0 [ 6.826126] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.831329] intel_pstate: CPU model not supported [ 6.835033] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.840902] hid: raw HID events driver (C) Jiri Kosina [ 6.844559] usbcore: registered new interface driver usbhid [ 6.849170] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.849432] usbhid: USB HID core driver [ 6.849700] drop_monitor: Initializing network drop monitor service [ 6.849867] Initializing XFRM netlink socket [ 6.850259] NET: Registered protocol family 10 [ 6.861967] Segment Routing with IPv6 [ 6.865789] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.872559] NET: Registered protocol family 17 [ 6.875939] mpls_gso: MPLS GSO support [ 6.884946] RAS: Correctable Errors collector initialized. [ 6.888383] AVX version of gcm_enc/dec engaged. [ 6.890783] AES CTR mode by8 optimization enabled [ 7.027644] sched_clock: Marking stable (7027546267, 0)->(7982095912, -954549645) [ 7.033393] registered taskstats version 1 [ 7.036515] Loading compiled-in X.509 certificates [ 7.039788] zswap: loaded using pool lzo/zbud [ 7.088608] Key type big_key registered [ 7.109856] Key type encrypted registered [ 7.112564] ima: No TPM chip found, activating TPM-bypass! [ 7.115582] ima: Allocated hash algorithm: sha1 [ 7.118202] ima: No architecture policies found [ 7.120635] evm: Initialising EVM extended attributes: [ 7.123857] evm: security.selinux [ 7.125730] evm: security.ima [ 7.127545] evm: security.capability [ 7.129554] evm: HMAC attrs: 0x1 [ 7.132989] rtc_cmos 00:05: setting system clock to 2026-09-16 17:09:03 UTC (1789578543) [ 7.142478] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 7.146448] debug: unmapping init [mem 0xffffffff96382000-0xffffffff96658fff] [ 7.162836] Write protecting the kernel read-only data: 28672k [ 7.174311] debug: unmapping init [mem 0xffffffff94a03000-0xffffffff94bfffff] [ 7.182161] debug: unmapping init [mem 0xffffffff95314000-0xffffffff953fffff] [ 7.251596] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.270066] systemd[1]: Detected virtualization kvm. [ 7.281430] systemd[1]: Detected architecture x86-64. [ 7.290519] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.354604] systemd[1]: No hostname configured. [ 7.358667] systemd[1]: Set hostname to . [ 7.367972] random: systemd: uninitialized urandom read (16 bytes read) [ 7.374535] systemd[1]: Initializing machine ID from random generator. [ 7.493135] random: ln: uninitialized urandom read (6 bytes read) [ 7.687855] random: systemd: uninitialized urandom read (16 bytes read) [ 7.691392] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.697760] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.711617] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.220478] device-mapper: uevent: version 1.0.3 [ 9.224215] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 10.407006] hrtimer: interrupt took 10998600 ns Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.088611] random: fast init done [ 11.173436] virtio_net virtio0 ens2: renamed from eth0 [ 11.176882] scsi host0: ata_piix [ 11.226354] scsi host1: ata_piix [ 11.228895] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.234247] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.648520] random: crng init done [ 16.652216] random: 7 urandom warning(s) missed due to ratelimiting [ 16.867272] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.082637] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.609870] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.354275] SELinux: Disabled at runtime. [ 22.461382] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.478712] systemd[1]: Detected virtualization kvm. [ 22.481945] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.125681] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.132710] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.147245] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.164183] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.175269] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.204746] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.238359] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 24.544794] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.672450] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.378557] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.468132] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.860543] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.975256] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 31.945240] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.780935] NFS: Registering the id_resolver key type [ 32.788849] Key type id_resolver registered [ 32.797813] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-server login: [ 97.491266] libcfs: loading out-of-tree module taints kernel. [ 97.529159] Key type ._llcrypt registered [ 97.530957] Key type .llcrypt registered [ 97.646980] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 119.397709] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 121.130665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.158076] alg: No test for adler32 (adler32-zlib) [ 123.498265] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 124.591756] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 126.472493] Key type lgssc registered [ 128.898630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.180134] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.405826] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 215.060730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.106535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.444625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.476944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.580548] Lustre: lustre-MDT0000: new disk, initializing [ 216.659803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.683829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.932089] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.261785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.381648] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 237.436256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 237.439931] Lustre: Skipped 1 previous similar message [ 237.532584] Lustre: lustre-MDT0001: new disk, initializing [ 237.637436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.668123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 237.675531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 242.751620] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.407408] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 259.871280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.183687] Lustre: lustre-OST0000: new disk, initializing [ 260.189995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.201449] Lustre: 8466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.304914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.283245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 265.304404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 265.381220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.251463] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.738193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.932848] Lustre: lustre-OST0001: new disk, initializing [ 285.936724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 285.945378] Lustre: 9540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 286.019516] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.952250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 292.972359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 293.060523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.519955] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 306.815632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 314.468951] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 322.328866] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.742672] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing yml_node [ 334.043480] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 337.876809] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 340.851404] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 342.603784] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 13:14:37 EDT 2026 [ 363.190642] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 365.314876] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 385.508627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.508905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.528994] Lustre: Skipped 3 previous similar messages [ 385.565861] Lustre: Skipped 3 previous similar messages [ 390.624944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.636946] Lustre: Skipped 3 previous similar messages [ 395.749856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 395.760634] Lustre: Skipped 3 previous similar messages [ 396.256141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 396.642209] Lustre: server umount lustre-MDT0000 complete [ 401.011296] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578937 with bad export cookie 15546644875426242172 [ 401.013174] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 401.025188] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 405.993143] LustreError: 6535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.006169] LustreError: 6535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 406.006964] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.016360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 406.027673] Lustre: Skipped 1 previous similar message [ 406.041138] Lustre: Skipped 1 previous similar message [ 407.014753] LustreError: 6537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.045061] LustreError: 6537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 407.657810] Lustre: server umount lustre-MDT0001 complete [ 412.879780] Lustre: server umount lustre-OST0000 complete [ 418.169877] Lustre: server umount lustre-OST0001 complete [ 426.923146] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 438.498691] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 482.334352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.578299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 482.600699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 482.691791] Lustre: lustre-MDT0000: new disk, initializing [ 482.808640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.841440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 488.702989] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 501.952640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.200975] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 502.313992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 502.320315] Lustre: Skipped 1 previous similar message [ 502.458886] Lustre: lustre-MDT0001: new disk, initializing [ 502.640543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 502.747490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 502.771623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 508.020459] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 519.281468] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 525.780914] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.025147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.326066] Lustre: lustre-OST0000: new disk, initializing [ 534.331482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 534.340978] Lustre: 17068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 534.456983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.873247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 535.883918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 536.029259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 540.314047] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 551.365794] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 557.585242] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 557.944960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 563.903213] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 564.132039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 566.754411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.768672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.781229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.792758] Lustre: Skipped 1 previous similar message [ 566.829086] Lustre: Skipped 3 previous similar messages [ 573.206296] Lustre: server umount lustre-OST0000 complete [ 579.555160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.561919] LustreError: Skipped 1 previous similar message [ 579.572095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.595837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.602418] Lustre: Skipped 2 previous similar messages [ 592.354187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 592.598710] Lustre: server umount lustre-MDT0000 complete [ 597.201456] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579133 with bad export cookie 15546644875426243705 [ 597.216551] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.219863] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 597.570993] Lustre: server umount lustre-MDT0001 complete [ 602.640946] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:18:57 (1789579137) [ 605.731984] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 607.803713] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:19:02 (1789579142) [ 615.495256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.108410] LustreError: 19479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.191868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.548975] LustreError: 19480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.692418] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 625.644886] LustreError: 19479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.638503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.607522] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 651.301073] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.927600] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 672.390863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 680.717170] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 691.664608] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 700.897441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 700.904675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.924195] Lustre: Skipped 1 previous similar message [ 700.940221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 700.955610] Lustre: Skipped 8 previous similar messages [ 707.059414] LustreError: 21305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.077320] LustreError: 21305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 707.338421] Lustre: server umount lustre-OST0000 complete [ 712.166088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.176522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 712.184054] Lustre: Skipped 1 previous similar message [ 712.198431] LustreError: Skipped 1 previous similar message [ 718.251762] Lustre: server umount lustre-MDT0000 complete [ 722.646034] LustreError: 21326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579259 with bad export cookie 15546644875426244580 [ 722.657620] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.662664] LustreError: 21326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 723.367656] Lustre: server umount lustre-MDT0001 complete [ 734.572437] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 738.115630] Key type lgssc unregistered [ 738.697634] LNet: 23016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.707149] LNetError: 23016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.744427] LNet: Removed LNI 192.168.204.152@tcp [ 740.290830] Key type .llcrypt unregistered [ 740.297443] Key type ._llcrypt unregistered [ 769.427572] Key type ._llcrypt registered [ 769.429162] Key type .llcrypt registered [ 769.558602] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 771.910815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 772.058239] alg: No test for adler32 (adler32-zlib) [ 773.162950] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 773.551265] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 775.352312] Key type lgssc registered [ 776.661802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 788.535825] loop: module loaded [ 788.683320] loop0: detected capacity change from 0 to 204800000 [ 789.021220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 789.037403] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 789.537417] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 796.749481] loop0: detected capacity change from 0 to 204800000 [ 797.054135] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 797.069311] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 797.089892] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 803.467316] loop0: detected capacity change from 0 to 204800000 [ 810.301355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 810.347722] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.726545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 811.805375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 811.984326] Lustre: lustre-MDT0000: new disk, initializing [ 812.127174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.166611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 818.439648] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 829.264778] loop1: detected capacity change from 0 to 204800000 [ 836.521557] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.786130] Lustre: lustre-OST07c6: new disk, initializing [ 836.791333] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 836.795370] Lustre: Skipped 1 previous similar message [ 836.800619] Lustre: 26203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 836.908764] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 845.428100] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 845.474470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 845.484086] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 845.534878] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 851.889763] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 851.901218] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 851.908140] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 851.914729] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 851.921464] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 851.928396] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 863.207065] Lustre: 26211:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 863.218590] Lustre: 26211:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 863.225798] Lustre: 26211:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 863.233444] Lustre: 26211:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 863.239529] Lustre: 26211:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 863.245723] Lustre: 26211:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 868.674784] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 868.686320] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 868.692503] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 868.697205] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.702880] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 868.709105] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.718390] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 868.728121] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.734130] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 868.740608] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.756511] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 868.769368] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.005124] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 872.020507] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 872.028242] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 872.035856] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.047134] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 872.058236] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.072070] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 872.081267] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.088249] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 872.094043] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.108380] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 872.122024] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 877.224095] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 877.235568] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 877.242052] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 877.250651] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 877.257518] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 877.266278] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 877.272060] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 877.289159] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 877.294569] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 877.302531] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 877.319669] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 877.345274] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.500426] Lustre: 26211:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 885.513077] Lustre: 26211:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 885.528114] Lustre: 26211:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 885.539188] Lustre: 26211:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 885.548101] Lustre: 26211:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 885.557631] Lustre: 26211:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 885.569417] Lustre: 26211:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 885.576142] Lustre: 26211:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 885.591883] Lustre: 26211:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 885.601502] Lustre: 26211:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 885.613751] Lustre: 26211:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 885.629408] Lustre: 26211:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 902.146471] Lustre: 26211:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 902.164517] Lustre: 26211:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 902.172621] Lustre: 26211:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 902.178981] Lustre: 26211:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 902.185445] Lustre: 26211:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 902.193716] Lustre: 26211:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 902.201423] Lustre: 26211:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 902.207319] Lustre: 26211:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 902.218821] Lustre: 26211:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 902.229876] Lustre: 26211:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 902.243371] Lustre: 26211:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 902.251379] Lustre: 26211:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 948.196114] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.202177] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.262224] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 951.586319] Lustre: server umount lustre-OST07c6 complete [ 964.120575] Lustre: server umount lustre-MDT0000 complete [ 982.582965] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 985.733379] Key type lgssc unregistered [ 986.109568] LNet: 28761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.121945] LNetError: 28761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.135489] LNet: Removed LNI 192.168.204.152@tcp [ 987.191827] Key type .llcrypt unregistered [ 987.203317] Key type ._llcrypt unregistered [ 1007.789403] Key type ._llcrypt registered [ 1007.792581] Key type .llcrypt registered [ 1007.988619] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1009.230853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.333041] alg: No test for adler32 (adler32-zlib) [ 1010.533534] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1010.749815] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1012.416255] Key type lgssc registered [ 1014.030212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1022.676882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1022.708870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.181770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.056246] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1033.704102] LustreError: 30029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.722972] LustreError: 30029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1038.823183] LustreError: 30030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.953361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.277579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1046.378570] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1055.382454] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.296630] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1070.944843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.592361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.404837] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1088.712143] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1094.218700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1109.477690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.487278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 1109.489215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1109.509883] Lustre: Skipped 1 previous similar message [ 1109.543996] LustreError: Skipped 1 previous similar message [ 1110.405226] Lustre: server umount lustre-OST0000 complete [ 1114.597111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.618421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.624177] Lustre: Skipped 2 previous similar messages [ 1119.713840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.160696] Lustre: server umount lustre-MDT0000 complete [ 1124.838518] LustreError: 30029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.868548] LustreError: 30029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1125.484192] LustreError: 30012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579661 with bad export cookie 2001344138308286218 [ 1125.485493] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.511029] LustreError: 30012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1125.910297] Lustre: server umount lustre-MDT0001 complete [ 1139.403512] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1142.778566] Key type lgssc unregistered [ 1143.181369] LNet: 33766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.203247] LNetError: 33766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.221553] LNet: Removed LNI 192.168.204.152@tcp [ 1144.290613] Key type .llcrypt unregistered [ 1144.292814] Key type ._llcrypt unregistered [ 1166.829649] Key type ._llcrypt registered [ 1166.833816] Key type .llcrypt registered [ 1166.986464] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1168.104068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1168.148086] alg: No test for adler32 (adler32-zlib) [ 1169.748579] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1170.148825] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1171.816273] Key type lgssc registered [ 1173.591594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.417383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.489250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.348607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.672210] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1198.571567] LustreError: 35037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.610426] LustreError: 35037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1202.885421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.211673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1206.861919] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1214.651669] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.127498] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1231.183093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.737354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.781960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1238.830171] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1247.843560] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1258.468209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1258.477036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.481584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.509948] Lustre: Skipped 1 previous similar message [ 1263.601805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.606057] Lustre: Skipped 2 previous similar messages [ 1264.533772] Lustre: server umount lustre-OST0000 complete [ 1269.733152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1269.741638] LustreError: Skipped 1 previous similar message [ 1269.750361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.768380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.831396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.841899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.103945] Lustre: server umount lustre-MDT0000 complete [ 1278.650484] LustreError: 35016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579815 with bad export cookie 17924657762050316365 [ 1278.656492] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.663384] LustreError: 35016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.932373] Lustre: server umount lustre-MDT0001 complete [ 1293.291253] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1297.571147] Key type lgssc unregistered [ 1298.059298] LNet: 38571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.081035] LNetError: 38571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.130454] LNet: Removed LNI 192.168.204.152@tcp [ 1299.643671] Key type .llcrypt unregistered [ 1299.651066] Key type ._llcrypt unregistered [ 1325.977076] Key type ._llcrypt registered [ 1325.981392] Key type .llcrypt registered [ 1326.227326] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1328.003342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.042643] alg: No test for adler32 (adler32-zlib) [ 1329.417689] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1329.749815] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1331.472235] Key type lgssc registered [ 1332.954697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.222953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.239400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.652880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.695113] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1353.202665] LustreError: 39987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.223212] LustreError: 39987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1358.306388] LustreError: 39988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.464489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.564429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.013763] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1378.144302] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1385.201036] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1392.818784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.367989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.987959] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1407.753307] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.861534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1423.335149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.339583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.362346] Lustre: Skipped 1 previous similar message [ 1423.389183] Lustre: Skipped 2 previous similar messages [ 1425.255135] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [ 1427.223120] Lustre: server umount lustre-MDT0000 complete [ 1428.469064] LustreError: 39988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.489686] LustreError: 39988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1433.571658] LustreError: 39988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.608730] LustreError: 39988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1433.743434] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579970 with bad export cookie 8090291216304232631 [ 1433.759172] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.776461] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1435.470336] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.52@tcp (stopping) [ 1438.692248] LustreError: 41411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.703996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.738782] Lustre: Skipped 1 previous similar message [ 1438.740604] LustreError: 41411:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1438.750132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1440.385188] Lustre: server umount lustre-MDT0001 complete [ 1456.947459] Lustre: server umount lustre-OST0000 complete [ 1478.734855] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1482.428882] Key type lgssc unregistered [ 1482.872884] LNet: 43867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.883286] LNetError: 43867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.945257] LNet: Removed LNI 192.168.204.152@tcp [ 1485.227556] Key type .llcrypt unregistered [ 1485.229899] Key type ._llcrypt unregistered [ 1512.525913] Key type ._llcrypt registered [ 1512.528152] Key type .llcrypt registered [ 1512.654287] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1514.030384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1514.264162] alg: No test for adler32 (adler32-zlib) [ 1515.341640] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1515.652354] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1517.344221] Key type lgssc registered [ 1518.564565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.991173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1527.026606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.644965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.644282] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1538.024369] LustreError: 45281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.036578] LustreError: 45281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1543.144849] LustreError: 45282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.577310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.067271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.780123] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1559.073623] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1565.319835] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1573.629904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.425465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.194596] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1590.394144] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1597.975195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1608.163487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.171149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1608.181474] Lustre: Skipped 1 previous similar message [ 1608.195687] Lustre: Skipped 1 previous similar message [ 1612.624365] Lustre: lustre-OST0000: Not available for connect from 192.168.204.52@tcp (stopping) [ 1614.415581] Lustre: server umount lustre-OST0000 complete [ 1633.764337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.776054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.784669] Lustre: Skipped 3 previous similar messages [ 1637.860942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.872858] Lustre: Skipped 1 previous similar message [ 1639.641233] Lustre: server umount lustre-MDT0000 complete [ 1642.986787] LustreError: 46571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1643.031159] LustreError: 46571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1643.550704] LustreError: 45260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580179 with bad export cookie 1419854316870284539 [ 1643.555267] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.572820] LustreError: 45260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1650.034627] Lustre: server umount lustre-MDT0001 complete [ 1661.844988] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1665.229621] Key type lgssc unregistered [ 1665.463584] LNet: 49017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.474480] LNetError: 49017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.539625] LNet: Removed LNI 192.168.204.152@tcp [ 1667.432127] Key type .llcrypt unregistered [ 1667.434947] Key type ._llcrypt unregistered [ 1701.296441] Key type ._llcrypt registered [ 1701.313506] Key type .llcrypt registered [ 1701.452778] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1702.951051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.240854] alg: No test for adler32 (adler32-zlib) [ 1704.591745] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1705.020471] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1706.690553] Key type lgssc registered [ 1708.354487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1716.930861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1716.973633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.370081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.017488] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1728.493637] LustreError: 50431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.512421] LustreError: 50431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1732.033720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.326506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1736.041314] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1744.406722] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1751.854042] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1760.058541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.569237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.636622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1768.212663] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1779.677901] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1786.127804] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1789.773535] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1801.604197] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1613h38m56s [ 1803.359608] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1807.192059] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [ 1807.843529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.848443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.853186] Lustre: Skipped 2 previous similar messages [ 1807.868315] Lustre: Skipped 2 previous similar messages [ 1808.932518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.944641] Lustre: Skipped 3 previous similar messages [ 1811.850707] Lustre: server umount lustre-MDT0000 complete [ 1813.298722] LustreError: 50427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.019345] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.040743] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1815.382137] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1819.443846] LustreError: 51675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.894869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.021973] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.400462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.455900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1824.427232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1828.715691] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1832.269812] Lustre: DEBUG MARKER: Wait for df (36695) ... [ 1833.683359] Lustre: DEBUG MARKER: done [ 1838.569543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1838.573234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.582975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.077160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1843.821742] Lustre: server umount lustre-OST0000 complete [ 1850.339496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1850.339788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.378897] Lustre: Skipped 1 previous similar message [ 1850.402386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.415982] Lustre: Skipped 4 previous similar messages [ 1854.895425] Lustre: server umount lustre-MDT0000 complete [ 1858.019767] LustreError: 51675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.050085] LustreError: 51675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1859.246909] LustreError: 53061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580395 with bad export cookie 12438323505906739873 [ 1859.263435] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.264356] LustreError: 53061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1859.787260] Lustre: server umount lustre-MDT0001 complete [ 1873.040908] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1875.869667] Key type lgssc unregistered [ 1876.219375] LNet: 55360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.225919] LNetError: 55360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.248822] LNet: Removed LNI 192.168.204.152@tcp [ 1877.143087] Key type .llcrypt unregistered [ 1877.146707] Key type ._llcrypt unregistered [ 1918.884416] Key type ._llcrypt registered [ 1918.886432] Key type .llcrypt registered [ 1918.946037] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1919.873489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1919.905586] alg: No test for adler32 (adler32-zlib) [ 1920.952879] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1921.158913] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1922.816264] Key type lgssc registered [ 1923.828105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.733646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.755723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.191264] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1933.207287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1933.415326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.374679] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1947.278749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.440530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1947.462524] Lustre: 57564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1947.664758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1951.730579] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1961.385392] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.024379] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.482229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.805822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1984.067496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.456418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1991.802143] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2000.767393] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2007.441169] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1613h42m22s [ 2009.264845] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2017.105807] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2019.739031] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2019.899120] LustreError: 57571:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2040.167680] Lustre: lustre-MDT0000: Client b0dd0b58-81d1-4f93-a252-2ce492ea57af (at 192.168.204.52@tcp) reconnecting [ 2059.913534] LustreError: 57571:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2062.929442] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2066.222537] Lustre: DEBUG MARKER: done [ 2073.573057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.592753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.596166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2073.610902] Lustre: Skipped 1 previous similar message [ 2073.623147] LustreError: Skipped 1 previous similar message [ 2078.501841] Lustre: server umount lustre-OST0000 complete [ 2083.817235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.824446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.835422] Lustre: Skipped 2 previous similar messages [ 2083.857662] Lustre: Skipped 1 previous similar message [ 2084.903411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.915707] Lustre: Skipped 1 previous similar message [ 2089.989850] Lustre: server umount lustre-MDT0000 complete [ 2094.423158] LustreError: 57557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580630 with bad export cookie 16397801216487908835 [ 2094.441449] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.868555] Lustre: server umount lustre-MDT0001 complete [ 2106.954740] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2110.302559] Key type lgssc unregistered [ 2110.659694] LNet: 61898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.664318] LNetError: 61898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.679518] LNet: Removed LNI 192.168.204.152@tcp [ 2111.472442] Key type .llcrypt unregistered [ 2111.475165] Key type ._llcrypt unregistered [ 2161.788542] Key type ._llcrypt registered [ 2161.798498] Key type .llcrypt registered [ 2162.114175] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2164.574096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.721900] alg: No test for adler32 (adler32-zlib) [ 2166.094614] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2166.582048] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2168.344260] Key type lgssc registered [ 2169.994787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.537410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2173.564962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.958423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2174.993029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2175.220725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.197054] Lustre: Failing over lustre-MDT0000 [ 2176.526872] Lustre: server umount lustre-MDT0000 complete [ 2183.543707] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:45:18 (1789580718) [ 2191.238912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.896416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.503869] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2205.725950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.802797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2205.823660] Lustre: 64695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2206.018101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2210.743032] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2220.858782] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.567619] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2235.764421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2235.925291] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2236.137382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.219898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2243.261958] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2251.644702] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2257.380543] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2267.620873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2267.630863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.678172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.131369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.067339] Lustre: server umount lustre-MDT0000 complete [ 2277.347639] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580813 with bad export cookie 18056165231822087706 [ 2277.357597] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.367103] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2277.717964] Lustre: server umount lustre-MDT0001 complete [ 2282.055945] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2288.991393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2293.731993] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580814/real 1789580814] req@0000000006939b86 x1876511374223232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789580830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2293.779885] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.813893] Lustre: Skipped 1 previous similar message [ 2297.824110] Lustre: 63616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580818/real 1789580818] req@000000005c4a3e57 x1876511374223360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789580834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2298.925648] Lustre: 63616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580819/real 1789580819] req@0000000001cffe43 x1876511374223488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789580835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2300.784684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.048282] LustreError: 68453:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2302.052520] LustreError: 63614:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d9038e1a x1876511374224256/t0(0) o250->MGC192.168.204.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2302.368969] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.387300] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2302.470463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.606613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2303.459318] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580823/real 1789580823] req@0000000086c1deb1 x1876511374223744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789580839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2304.546536] LustreError: 66543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.566912] LustreError: 66543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2307.791827] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2309.600262] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580829/real 1789580829] req@00000000d97a6a69 x1876511374223872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789580845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2309.603576] LustreError: 68468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.634429] LustreError: 68468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2314.725020] LustreError: 68467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.739364] LustreError: 68467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2318.062453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.560931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2322.667922] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2322.855511] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2331.516823] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2338.517382] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2348.514813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.544131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.551985] Lustre: Skipped 5 previous similar messages [ 2352.509435] Lustre: server umount lustre-MDT0000 complete [ 2353.634058] LustreError: 68468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.644567] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580893 with bad export cookie 18056165231822089981 [ 2356.649469] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2356.666609] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2357.023716] Lustre: server umount lustre-MDT0001 complete [ 2363.915570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2374.815866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.136644] Lustre: 63617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580895/real 1789580895] req@000000004d2acf20 x1876511374253568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789580911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2375.165906] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.175317] Lustre: Skipped 2 previous similar messages [ 2382.691124] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.711190] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2382.781522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.838791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2387.549897] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2397.753493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.326781] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2403.330271] Lustre: Skipped 1 previous similar message [ 2403.499535] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2404.332758] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2413.768792] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2420.578965] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2429.923728] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.929949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.947526] Lustre: Skipped 2 previous similar messages [ 2429.961119] Lustre: Skipped 1 previous similar message [ 2435.043895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.050123] Lustre: Skipped 3 previous similar messages [ 2435.544038] Lustre: server umount lustre-MDT0000 complete [ 2440.171522] LustreError: 71150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.208910] LustreError: 71150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2440.427559] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580976 with bad export cookie 18056165231822091360 [ 2440.437287] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.444230] LustreError: 64687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2440.847800] Lustre: server umount lustre-MDT0001 complete [ 2448.518751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2453.083411] Lustre: DEBUG MARKER: files compared the same [ 2465.052300] Lustre: server umount lustre-OST0000 complete [ 2484.186665] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2487.494233] Key type lgssc unregistered [ 2487.821759] LNet: 74425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2487.829715] LNetError: 74425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.843758] LNet: Removed LNI 192.168.204.152@tcp [ 2488.731584] Key type .llcrypt unregistered [ 2488.733162] Key type ._llcrypt unregistered [ 2518.722972] Key type ._llcrypt registered [ 2518.734246] Key type .llcrypt registered [ 2518.958617] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2521.276829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.321597] alg: No test for adler32 (adler32-zlib) [ 2523.100824] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2523.754904] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2525.648346] Key type lgssc registered [ 2528.300053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.751479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2539.779784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.190303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.096179] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug +malloc all [ 2550.765217] LustreError: 75840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.795960] LustreError: 75840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2554.849820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.257864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.373207] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug +malloc all [ 2572.570086] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2582.030497] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2593.439280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.367450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.180843] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug +malloc all [ 2610.723230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2614.692662] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2626.025895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.028368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.044078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.051387] Lustre: Skipped 1 previous similar message [ 2629.839798] Lustre: server umount lustre-OST0000 complete [ 2636.258132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.267196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.273358] Lustre: Skipped 1 previous similar message [ 2636.282052] Lustre: Skipped 1 previous similar message [ 2640.588827] Lustre: server umount lustre-MDT0000 complete [ 2641.383411] LustreError: 75840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.420490] LustreError: 75840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2644.702818] LustreError: 75820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581181 with bad export cookie 13877213276681307063 [ 2644.706512] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2644.718656] LustreError: 75820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2645.131420] Lustre: server umount lustre-MDT0001 complete [ 2656.223805] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2659.311394] Key type lgssc unregistered [ 2659.578133] LNet: 79372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.591509] LNetError: 79372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.615066] LNet: Removed LNI 192.168.204.152@tcp [ 2660.561993] Key type .llcrypt unregistered [ 2660.568537] Key type ._llcrypt unregistered [ 2687.267542] Key type ._llcrypt registered [ 2687.274916] Key type .llcrypt registered [ 2687.457579] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2688.769804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.923234] alg: No test for adler32 (adler32-zlib) [ 2690.208272] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2690.662813] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2692.456438] Key type lgssc registered [ 2694.066680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2705.291724] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2705.381816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.185053] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2726.304488] LustreError: 80761:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 2753.056154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.462694] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2770.746631] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2780.255674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.540351] Lustre: *** cfs_fail_loc=706, val=0*** [ 2781.073742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.155895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2786.051484] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2786.274948] LustreError: 80786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.291915] LustreError: 80786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2788.327255] LustreError: 81847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.337528] LustreError: 81847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2793.448279] LustreError: 81846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.468689] LustreError: 81846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2796.978969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.405539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2803.367748] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2813.953285] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2820.865790] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2827.244537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2827.259964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.276564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.268854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.293762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.295712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.335613] Lustre: Skipped 1 previous similar message [ 2831.636843] Lustre: server umount lustre-OST0000 complete [ 2843.620134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.622345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.648392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.654184] Lustre: Skipped 1 previous similar message [ 2845.095702] Lustre: server umount lustre-MDT0000 complete [ 2847.780461] LustreError: 81851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.815478] LustreError: 81851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2849.914444] LustreError: 80767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581386 with bad export cookie 3007492850339552829 [ 2849.923957] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.939862] LustreError: 80767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2850.214177] Lustre: server umount lustre-MDT0001 complete [ 2863.391371] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2867.573733] Key type lgssc unregistered [ 2868.362732] LNet: 84380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.382411] LNetError: 84380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2868.419762] LNet: Removed LNI 192.168.204.152@tcp [ 2870.576692] Key type .llcrypt unregistered [ 2870.578779] Key type ._llcrypt unregistered [ 2906.155917] Key type ._llcrypt registered [ 2906.160891] Key type .llcrypt registered [ 2906.364712] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2907.936211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2908.066387] alg: No test for adler32 (adler32-zlib) [ 2909.304605] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2909.592400] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2911.288472] Key type lgssc registered [ 2913.091621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2923.040151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2923.072336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.803850] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2939.029786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.901197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.012360] LustreError: 86483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.042036] LustreError: 86483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2946.293576] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2950.115081] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.143883] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2955.242968] LustreError: 86486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.680039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2959.202972] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.234464] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2959.265554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.470936] LustreError: 86487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.495793] LustreError: 86487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2966.426491] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2975.719177] LustreError: 86487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.745701] LustreError: 86487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2977.821230] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.836501] Lustre: MGS: Client 95307c5f-d4dd-43a6-997b-2223144bf40a (at 0@lo) reconnecting [ 2977.864285] LustreError: 86498:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2978.212246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.316277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2984.943394] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2998.760226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.764481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2998.776031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.780103] Lustre: Skipped 1 previous similar message [ 3003.885773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.892061] Lustre: Skipped 1 previous similar message [ 3005.127241] Lustre: server umount lustre-OST0000 complete [ 3014.116147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3014.119883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.133503] Lustre: Skipped 1 previous similar message [ 3014.145349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.151673] Lustre: Skipped 1 previous similar message [ 3016.487898] Lustre: server umount lustre-MDT0000 complete [ 3019.238144] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.251241] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3022.244479] Lustre: server umount lustre-MDT0001 complete [ 3027.706604] Lustre: server umount MGS complete [ 3040.961944] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:59:35 (1789581575) [ 3084.864775] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 3096.554796] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3140.690141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.638777] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3158.240279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.440721] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3158.482627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3158.556624] Lustre: lustre-MDT0001: new disk, initializing [ 3158.706575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3158.760292] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3158.777682] Lustre: 93672:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3158.803100] LustreError: 93672:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3164.954758] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3179.007859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3179.214952] Lustre: lustre-OST0000: new disk, initializing [ 3179.218471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3179.222290] Lustre: 94440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3179.288880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.743128] Lustre: 94727:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001c3ff114 x1876512153316224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789581739 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3187.786114] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3199.753979] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3207.137458] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3210.011258] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.025951] Lustre: MGS: Client bea1d19d-dc0c-4e3f-8cc6-dfc2201a507e (at 0@lo) reconnecting [ 3210.093159] Lustre: 92939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3210.128531] LustreError: 93658:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3210.136591] LustreError: 93658:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3210.154316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3210.298838] Lustre: lustre-MDT0000: new disk, initializing [ 3210.411786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.435924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3211.491302] Lustre: 95511:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000105c27e0 x1876512153323648/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789581763 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3215.236850] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3216.546466] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3220.591928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3220.604996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3220.702263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3246.564351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.569548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.590246] Lustre: Skipped 1 previous similar message [ 3246.597798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3252.472160] Lustre: server umount lustre-OST0000 complete [ 3259.365420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.381273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.408338] Lustre: Skipped 2 previous similar messages [ 3259.416356] Lustre: Skipped 3 previous similar messages [ 3264.812259] Lustre: server umount lustre-MDT0000 complete [ 3269.605462] LustreError: 93642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.645561] LustreError: 93642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3276.311459] Lustre: server umount lustre-MDT0001 complete [ 3281.712633] Lustre: server umount MGS complete [ 3296.669227] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:03:50 (1789581830) [ 3321.253505] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3324.900659] Key type lgssc unregistered [ 3325.129840] LNet: 98010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3325.153757] LNetError: 98010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3325.171748] LNet: Removed LNI 192.168.204.152@tcp [ 3326.339975] Key type .llcrypt unregistered [ 3326.343279] Key type ._llcrypt unregistered [ 3344.949639] Key type ._llcrypt registered [ 3344.952226] Key type .llcrypt registered [ 3345.103180] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3346.143321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3346.174053] alg: No test for adler32 (adler32-zlib) [ 3347.153439] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3347.330292] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3348.936304] Key type lgssc registered [ 3350.398963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3359.844691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3359.890713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.543459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.168499] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3370.984472] LustreError: 99226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.019786] LustreError: 99226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3376.099553] LustreError: 99227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.613170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.875882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.255715] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3397.919282] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3404.567892] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3412.949191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3413.359679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.149320] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3426.653126] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3430.777947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3441.123359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3441.129627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.154114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3442.149367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3442.160563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.184706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3444.212987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3444.240165] Lustre: Skipped 1 previous similar message [ 3446.936569] Lustre: server umount lustre-OST0000 complete [ 3452.393301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3452.402381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.411272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.437546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.804879] Lustre: server umount lustre-MDT0000 complete [ 3459.555543] LustreError: 100152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.582503] LustreError: 100152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3461.989480] LustreError: 99206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581998 with bad export cookie 8112588716206337433 [ 3461.992812] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.001812] LustreError: 99206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3462.365458] Lustre: server umount lustre-MDT0001 complete [ 3475.578246] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3479.032641] Key type lgssc unregistered [ 3479.303812] LNet: 102785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3479.309714] LNetError: 102785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3479.333135] LNet: Removed LNI 192.168.204.152@tcp [ 3480.006668] Key type .llcrypt unregistered [ 3480.009298] Key type ._llcrypt unregistered [ 3505.575300] Key type ._llcrypt registered [ 3505.578242] Key type .llcrypt registered [ 3505.645485] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3506.844299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3506.989913] alg: No test for adler32 (adler32-zlib) [ 3508.448563] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3508.848922] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3510.576312] Key type lgssc registered [ 3511.907602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3519.223962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3519.235289] LustreError: 103860:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3519.265647] LustreError: 103867:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3519.269979] LustreError: 103860:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3519.275265] LustreError: 103860:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3519.285834] LustreError: 103860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3519.289302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.027599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.049617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.520913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3535.335586] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3541.383898] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing lsmod [ 3544.260187] LustreError: 105074:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3544.263208] LustreError: 105073:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3544.268838] LustreError: 105074:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3544.274942] LustreError: 105074:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3544.281163] LustreError: 105074:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3544.288928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.604884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.700438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3551.211139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3551.227828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.248885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3553.695455] Lustre: server umount lustre-MDT0000 complete [ 3554.795893] LustreError: 103902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.824037] LustreError: 103902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3557.894492] LustreError: 103883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582094 with bad export cookie 11295840956925507677 [ 3557.912667] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3559.008826] LustreError: 104336:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000033931caf x1876512782439424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3559.253771] Lustre: server umount lustre-MDT0001 complete [ 3574.284189] Lustre: server umount lustre-OST0000 complete [ 3583.221212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.750290] LustreError: 106136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.875102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.595576] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3589.092917] LustreError: 106137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.210159] LustreError: 106136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.379611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.844785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.438313] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3611.511672] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3619.085657] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3626.594863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.130975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.188195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3633.243517] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3640.922889] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3654.122931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3654.125094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.127277] LustreError: Skipped 1 previous similar message [ 3654.128617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3654.137199] Lustre: Skipped 3 previous similar messages [ 3654.146667] Lustre: Skipped 3 previous similar messages [ 3657.192157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3657.199932] Lustre: Skipped 1 previous similar message [ 3660.015482] Lustre: server umount lustre-OST0000 complete [ 3664.354904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3664.377104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.405020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3667.435515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.503383] Lustre: server umount lustre-MDT0000 complete [ 3672.546179] LustreError: 106137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.577292] LustreError: 106137:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3675.390742] LustreError: 106117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582211 with bad export cookie 11295840956925508608 [ 3675.400514] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3675.410966] LustreError: 106117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3676.154930] Lustre: server umount lustre-MDT0001 complete [ 3688.401842] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3691.382926] Key type lgssc unregistered [ 3691.786412] LNet: 109691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3691.807642] LNetError: 109691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3692.842712] LNet: Removed LNI 192.168.204.152@tcp [ 3693.977709] Key type .llcrypt unregistered [ 3693.979933] Key type ._llcrypt unregistered [ 3719.182900] Key type ._llcrypt registered [ 3719.185649] Key type .llcrypt registered [ 3719.282056] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3720.269805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.445044] alg: No test for adler32 (adler32-zlib) [ 3721.567312] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3721.803212] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3723.512173] Key type lgssc registered [ 3724.957066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3733.521472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3733.538768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.183506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.072170] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3744.748971] LustreError: 111105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.769241] LustreError: 111105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3749.861982] LustreError: 111106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.805617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.255901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3757.565028] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3766.323864] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3773.038980] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3780.951539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.330948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.022572] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3798.418350] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3804.922412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3808.966410] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3813.117618] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3820.513135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.517460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.534738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.557454] Lustre: Skipped 1 previous similar message [ 3820.571627] Lustre: Skipped 1 previous similar message [ 3825.640271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3825.655668] Lustre: Skipped 1 previous similar message [ 3830.755254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.289960] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3832.487513] Lustre: server umount lustre-OST0000 complete [ 3839.462446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3839.471841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.503623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.515809] Lustre: Skipped 1 previous similar message [ 3840.999909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.118682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.134700] Lustre: Skipped 3 previous similar messages [ 3851.233903] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3851.471810] Lustre: server umount lustre-MDT0000 complete [ 3855.636149] LustreError: 111087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582392 with bad export cookie 12011163278578648899 [ 3855.636414] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3855.643244] LustreError: 111087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3855.910942] Lustre: server umount lustre-MDT0001 complete [ 3871.294993] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3875.259394] Key type lgssc unregistered [ 3875.668274] LNet: 114758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3875.677665] LNetError: 114758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3875.704615] LNet: Removed LNI 192.168.204.152@tcp [ 3876.920227] Key type .llcrypt unregistered [ 3876.924085] Key type ._llcrypt unregistered [ 3900.443257] Key type ._llcrypt registered [ 3900.447784] Key type .llcrypt registered [ 3900.592814] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3901.588677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3901.731061] alg: No test for adler32 (adler32-zlib) [ 3902.787784] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3903.017808] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3904.680230] Key type lgssc registered [ 3905.972557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.487554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.528880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.972791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.506747] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3926.503761] LustreError: 115976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.528583] LustreError: 115976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3930.058806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.597152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.559655] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3945.136514] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3951.536922] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3960.301811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.645777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.689142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3967.974078] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3977.696134] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.426255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.436317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.450043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.507249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.508213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.546455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3993.249664] Lustre: server umount lustre-OST0000 complete [ 4000.747872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.766544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.777345] Lustre: Skipped 1 previous similar message [ 4000.800750] Lustre: Skipped 2 previous similar messages [ 4004.866404] Lustre: server umount lustre-MDT0000 complete [ 4005.857522] LustreError: 115972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.874303] LustreError: 115972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4009.372357] LustreError: 115957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582545 with bad export cookie 9675085941899228899 [ 4009.377630] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.379162] LustreError: 115957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4009.825096] Lustre: server umount lustre-MDT0001 complete [ 4028.407975] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 4032.984960] Key type lgssc unregistered [ 4033.282176] LNet: 119532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4033.294476] LNetError: 119532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4033.322157] LNet: Removed LNI 192.168.204.152@tcp [ 4034.118901] Key type .llcrypt unregistered [ 4034.120663] Key type ._llcrypt unregistered [ 4061.766266] Key type ._llcrypt registered [ 4061.768259] Key type .llcrypt registered [ 4061.910873] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4063.502994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4063.533081] alg: No test for adler32 (adler32-zlib) [ 4064.579278] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 4064.892492] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 4066.571090] Key type lgssc registered [ 4068.316664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4077.253260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4077.280906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.893253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.258478] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4089.325093] LustreError: 120947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.340649] LustreError: 120947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4092.869360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.619689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4098.992866] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4108.444697] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4115.385684] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4123.576552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.965362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.999616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4130.599435] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4138.598257] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4145.345081] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4149.545317] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4149.550478] Lustre: Skipped 2 previous similar messages [ 4152.383892] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4152.411808] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4154.046504] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4154.053224] Lustre: Skipped 2 previous similar messages [ 4161.141340] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4161.149291] Lustre: Skipped 1 previous similar message [ 4168.771282] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4168.780664] Lustre: Skipped 2 previous similar messages [ 4169.366570] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.52@tcp 192.168.0.[2,10]@tcp [ 4181.340603] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.52/24@tcp in log lustre-MDT0000 [ 4181.348615] Lustre: Skipped 2 previous similar messages [ 4190.307675] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.52/24@tcp [ 4190.310886] Lustre: Skipped 1 previous similar message [ 4196.835448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.845363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.866092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.645259] Lustre: server umount lustre-OST0000 complete [ 4202.827497] Lustre: server umount lustre-MDT0000 complete [ 4205.027059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.041854] LustreError: 120941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.057402] LustreError: 120941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4206.915068] LustreError: 123984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582743 with bad export cookie 2148876167713692923 [ 4206.917988] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4206.953167] LustreError: 123984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4217.653538] Lustre: server umount lustre-MDT0001 complete [ 4231.081709] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 4233.748290] Key type lgssc unregistered [ 4234.015227] LNet: 125892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4234.027440] LNetError: 125892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4234.043762] LNet: Removed LNI 192.168.204.152@tcp [ 4234.993390] Key type .llcrypt unregistered [ 4234.996748] Key type ._llcrypt unregistered [ 4268.630723] Key type ._llcrypt registered [ 4268.633244] Key type .llcrypt registered [ 4268.803588] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4269.810121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4269.832315] alg: No test for adler32 (adler32-zlib) [ 4270.857264] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 4271.078599] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 4272.808219] Key type lgssc registered [ 4273.971643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4282.132495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4282.173683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.620267] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4283.644253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.674807] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.52/24@tcp [ 4288.180109] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4293.094890] LustreError: 127450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.107310] LustreError: 127450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4296.853978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.235285] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4297.292552] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.52/24@tcp [ 4297.371205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4303.130874] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4312.998293] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4319.774366] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4327.845520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.457338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.573787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4336.429720] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4346.730170] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4357.091473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4357.097879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4357.099348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4357.128373] Lustre: Skipped 1 previous similar message [ 4362.213371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4362.226306] Lustre: Skipped 2 previous similar messages [ 4363.228230] Lustre: server umount lustre-OST0000 complete [ 4368.866426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4368.876950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.889549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.449831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.452858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.151618] Lustre: server umount lustre-MDT0000 complete [ 4377.569567] LustreError: 127446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.595684] LustreError: 127446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4378.671824] LustreError: 127433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582915 with bad export cookie 2464348594863268381 [ 4378.678686] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.682732] LustreError: 127433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4379.013874] Lustre: server umount lustre-MDT0001 complete [ 4392.272489] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 4396.266311] Key type lgssc unregistered [ 4396.712461] LNet: 131042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4396.726822] LNetError: 131042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4396.748871] LNet: Removed LNI 192.168.204.152@tcp [ 4397.864752] Key type .llcrypt unregistered [ 4397.868478] Key type ._llcrypt unregistered [ 4427.087505] Key type ._llcrypt registered [ 4427.090958] Key type .llcrypt registered [ 4427.205813] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 4444.717867] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4445.578775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4445.688599] alg: No test for adler32 (adler32-zlib) [ 4446.744589] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 4447.057963] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 4448.800654] Key type lgssc registered [ 4449.892144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4488.171900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4488.220086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.535241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4489.603396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4489.779446] Lustre: lustre-MDT0000: new disk, initializing [ 4489.925385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4489.952500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4495.195942] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4507.240504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.387634] Lustre: 134441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4507.489531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4507.500353] Lustre: Skipped 1 previous similar message [ 4507.582855] Lustre: lustre-MDT0001: new disk, initializing [ 4507.692859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4507.735030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4507.757173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4512.742342] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4523.574354] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4530.855685] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4538.378996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.645983] Lustre: lustre-OST0000: new disk, initializing [ 4538.650371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4538.658985] Lustre: 136408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4538.749565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.572433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4544.584377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4544.708981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4545.103410] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4557.862108] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4564.529364] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4564.929809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4570.859041] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4571.073878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4575.718063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4575.724115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.725820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4575.742397] Lustre: Skipped 1 previous similar message [ 4575.749480] Lustre: Skipped 1 previous similar message [ 4579.638873] Lustre: server umount lustre-OST0000 complete [ 4584.418396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4584.436138] LustreError: Skipped 1 previous similar message [ 4584.450919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.464044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4585.955705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.958578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4585.982578] Lustre: Skipped 1 previous similar message [ 4590.676597] Lustre: server umount lustre-MDT0000 complete [ 4595.462264] LustreError: 134433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789583131 with bad export cookie 3151094722108562916 [ 4595.472340] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4595.709697] Lustre: server umount lustre-MDT0001 complete [ 4605.163642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.677175] LustreError: 138628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.701636] LustreError: 138628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4605.769937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.050617] LustreError: 138629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.156514] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4616.174588] LustreError: 138628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.629854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.021854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4626.188759] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4636.003349] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4644.159545] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4653.629752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.109895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.924624] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4672.583994] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4677.799677] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4677.950828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4682.869965] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4683.183710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4692.841042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.080443] Lustre: lustre-OST0001: new disk, initializing [ 4693.083551] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4693.088305] Lustre: 142002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4693.177808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4700.667771] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4701.337804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4701.351619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4701.433473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4710.333465] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4710.608137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4717.061740] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4717.383295] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4724.732911] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff949ac46a1000.ost_server_uuid 50 [ 4726.488088] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff949ac46a1000.ost_server_uuid in IDLE state after 0 sec [ 4737.508360] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4737.522877] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.550894] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4737.557893] Lustre: Skipped 1 previous similar message [ 4739.376791] Lustre: server umount lustre-OST0001 complete [ 4742.644055] LustreError: 140453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4742.663211] LustreError: 140453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4747.748832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.754984] LustreError: 140452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.755923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4747.772622] Lustre: Skipped 2 previous similar messages [ 4747.791611] LustreError: 140452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4750.252789] Lustre: server umount lustre-OST0000 complete [ 4754.405916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4754.416085] LustreError: Skipped 1 previous similar message [ 4757.989109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4757.994765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.010028] Lustre: Skipped 2 previous similar messages [ 4758.030739] Lustre: Skipped 1 previous similar message [ 4760.968672] Lustre: server umount lustre-MDT0000 complete [ 4763.121570] LustreError: 138629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.159568] LustreError: 138629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4765.180227] LustreError: 138610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789583301 with bad export cookie 3151094722108563798 [ 4765.189977] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4765.202175] LustreError: 138610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4765.688540] Lustre: server umount lustre-MDT0001 complete [ 4792.313491] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 4796.330673] Key type lgssc unregistered [ 4796.772960] LNet: 145006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4796.787880] LNetError: 145006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4796.827384] LNet: Removed LNI 192.168.204.152@tcp [ 4798.014816] Key type .llcrypt unregistered [ 4798.016622] Key type ._llcrypt unregistered [ 4846.108648] Key type ._llcrypt registered [ 4846.113264] Key type .llcrypt registered [ 4846.220802] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 4864.311364] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4865.452492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4865.480800] alg: No test for adler32 (adler32-zlib) [ 4866.970558] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 4867.399724] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 4869.116265] Key type lgssc registered [ 4870.795846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4917.119714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4917.162117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.612380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4918.641673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4918.782302] Lustre: lustre-MDT0000: new disk, initializing [ 4918.882886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.909499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4924.572983] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4936.973491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.103733] Lustre: 149162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4937.193156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4937.198278] Lustre: Skipped 1 previous similar message [ 4937.326351] Lustre: lustre-MDT0001: new disk, initializing [ 4937.360123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4937.485484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4937.563068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4937.586991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4944.100742] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4956.514476] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4964.536936] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4973.137073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4973.522370] Lustre: lustre-OST0000: new disk, initializing [ 4973.525579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4973.535734] Lustre: 151132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4973.683361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4979.309036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4979.332509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4979.494519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4982.195329] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4994.342892] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5000.257682] Lustre: Failing over lustre-OST0000 [ 5002.212456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5002.224921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.236835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.548468] Lustre: server umount lustre-OST0000 complete [ 5020.643616] Lustre: 147383:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789583541/real 1789583541] req@00000000fb4f63e4 x1876514206421248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789583557 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5020.675803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.010543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5022.349396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.364425] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5025.748043] Lustre: 147383:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789583546/real 1789583546] req@00000000c9237239 x1876514206423040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789583562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5028.144663] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5029.233761] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5032.480139] Lustre: 147383:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789583551/real 1789583551] req@000000004616c52d x1876514206424064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789583567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5033.597326] Lustre: Failing over lustre-MDT0000 [ 5033.924545] Lustre: server umount lustre-MDT0000 complete [ 5034.977467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5034.990947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.537866] Lustre: 147382:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789583557/real 1789583557] req@00000000a24d42af x1876514206425088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789583573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5041.642836] LustreError: 149169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.664207] LustreError: 149169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5043.508292] LustreError: 149168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5046.753465] LustreError: 149172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5046.780542] LustreError: 149172:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5051.879686] LustreError: 150165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.911926] LustreError: 150165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5052.896808] Lustre: 147381:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789583572/real 1789583572] req@000000003105612f x1876514206434048/t0(0) o400->MGC192.168.204.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789583588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5052.945630] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5053.829692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.021394] LustreError: 149167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.056595] LustreError: 149167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5063.138365] LustreError: 147379:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d094412a x1876514206438016/t0(0) o250->MGC192.168.204.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5063.415847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 5063.467453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.543023] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5063.987711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5068.808312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5068.843196] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5068.944109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5069.053107] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5079.521431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5079.527332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.531126] LustreError: Skipped 1 previous similar message [ 5079.533138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5079.543283] Lustre: Skipped 3 previous similar messages [ 5079.554344] Lustre: Skipped 1 previous similar message [ 5084.130641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5084.143199] Lustre: Skipped 1 previous similar message [ 5085.890944] Lustre: server umount lustre-OST0000 complete [ 5094.370775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.400030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.411995] Lustre: Skipped 1 previous similar message [ 5096.845255] Lustre: server umount lustre-MDT0000 complete [ 5099.489183] LustreError: 149173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5099.524450] LustreError: 149173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5100.860291] LustreError: 149152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789583637 with bad export cookie 13863819141750873914 [ 5100.863471] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.867264] LustreError: 149152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5101.201036] Lustre: server umount lustre-MDT0001 complete [ 5115.112755] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 5117.872460] Key type lgssc unregistered [ 5118.235545] LNet: 154793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5118.242544] LNetError: 154793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5119.272977] LNet: Removed LNI 192.168.204.152@tcp [ 5120.117888] Key type .llcrypt unregistered [ 5120.119937] Key type ._llcrypt unregistered [ 5161.551246] Key type ._llcrypt registered [ 5161.553824] Key type .llcrypt registered [ 5161.666196] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5162.422733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5162.457320] alg: No test for adler32 (adler32-zlib) [ 5163.622922] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 5164.009296] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 5165.816283] Key type lgssc registered [ 5167.458395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5178.553402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5178.571617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.192984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.371369] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5190.631414] LustreError: 156312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.682192] LustreError: 156312:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5195.751867] LustreError: 156313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.182772] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5200.866854] LustreError: 156312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.914799] LustreError: 156341:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5200.922476] LustreError: 156341:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5200.928648] LustreError: 156341:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5201.214140] Lustre: server umount lustre-MDT0000 complete [ 5219.562881] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 5230.675214] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5275.607547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.873696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5275.916471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5275.989528] Lustre: lustre-MDT0000: new disk, initializing [ 5276.069196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.085190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5281.113290] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5294.077979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.235108] Lustre: 159646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5294.342363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5294.356289] Lustre: Skipped 1 previous similar message [ 5294.424794] Lustre: lustre-MDT0001: new disk, initializing [ 5294.547896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5294.619911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5294.630515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5300.128990] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5310.740815] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5319.042570] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5327.611363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5327.926377] Lustre: lustre-OST0000: new disk, initializing [ 5327.930755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5327.945853] Lustre: 161614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5328.118950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.642112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5335.659165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5335.817969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5335.857466] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5347.689093] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5352.018827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5354.226817] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:38:08 (1789583888) [ 5647.840994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5647.854302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.872758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.359643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.395905] Lustre: Skipped 1 previous similar message [ 5648.404750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.422442] Lustre: Skipped 2 previous similar messages [ 5650.941920] Lustre: server umount lustre-MDT0000 complete [ 5653.479841] LustreError: 159657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.513589] LustreError: 159657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5655.438913] LustreError: 163379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789584191 with bad export cookie 16088732836415580342 [ 5655.447596] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.453378] LustreError: 163379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5658.593115] LustreError: 163381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.593782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.635508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5660.640735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5661.773676] Lustre: server umount lustre-MDT0001 complete [ 5672.483912] Lustre: server umount lustre-OST0000 complete [ 5682.660803] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 5692.320631] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5734.263601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.536795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5734.563479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5734.647383] Lustre: lustre-MDT0000: new disk, initializing [ 5734.717026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.730879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5739.907873] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5751.489726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.599134] Lustre: 166292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5751.674937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5751.682643] Lustre: Skipped 1 previous similar message [ 5751.757327] Lustre: lustre-MDT0001: new disk, initializing [ 5751.879697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5751.930213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5751.945076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5756.810959] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5767.859122] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5774.785721] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5782.419226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.755530] Lustre: lustre-OST0000: new disk, initializing [ 5782.761418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5782.774812] Lustre: 168260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5782.881850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5789.910555] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5792.347874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5792.363690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5792.484674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5801.509830] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5807.488821] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5807.819900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5813.077773] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5813.371753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5818.364252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.365173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5818.371650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.380412] Lustre: Skipped 1 previous similar message [ 5818.408265] LustreError: Skipped 1 previous similar message [ 5821.733150] Lustre: server umount lustre-OST0000 complete [ 5828.585958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5828.594700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.610321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5828.624453] Lustre: Skipped 1 previous similar message [ 5828.638321] Lustre: Skipped 2 previous similar messages [ 5832.382545] Lustre: server umount lustre-MDT0000 complete [ 5833.697287] LustreError: 167416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.722133] LustreError: 167416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5836.353216] LustreError: 166282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789584372 with bad export cookie 16088732836415739172 [ 5836.363131] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.364242] LustreError: 166282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5836.732661] Lustre: server umount lustre-MDT0001 complete [ 5846.481145] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:46:21 (1789584381) [ 5864.036449] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 5874.086436] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5913.977990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.183923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5914.209659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5914.309602] Lustre: lustre-MDT0000: new disk, initializing [ 5914.396662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5914.419285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5919.236892] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5930.377068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.573962] Lustre: 173017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5931.064279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5936.358586] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5946.539385] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5952.721835] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5959.881599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5960.179686] Lustre: lustre-OST0000: new disk, initializing [ 5960.182860] Lustre: Skipped 1 previous similar message [ 5960.186299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5960.190082] Lustre: Skipped 2 previous similar messages [ 5960.195166] Lustre: 174996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5967.095532] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5967.502544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5967.513417] Lustre: Skipped 1 previous similar message [ 5967.523709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5967.625756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5978.132745] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5988.332269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.338920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.348788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.370754] LustreError: Skipped 1 previous similar message [ 5991.778750] Lustre: server umount lustre-OST0000 complete [ 5997.548864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6003.034417] Lustre: server umount lustre-MDT0000 complete [ 6003.682183] LustreError: 173031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6003.699177] LustreError: 173031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6007.142268] LustreError: 173010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789584543 with bad export cookie 16088732836415740054 [ 6007.158215] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6007.165553] LustreError: 173010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6007.497600] Lustre: server umount lustre-MDT0001 complete [ 6017.370252] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:49:11 (1789584551) [ 6037.462623] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 6048.163980] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6090.610879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.970429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6091.029912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6091.141103] Lustre: lustre-MDT0000: new disk, initializing [ 6091.289457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.304919] Lustre: Skipped 2 previous similar messages [ 6091.350454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6096.508621] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6107.552617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.732992] Lustre: 179320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6108.354132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6113.851111] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6125.909994] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6133.650541] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6141.540052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6141.880062] Lustre: 181300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6148.174718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6148.378590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6149.757745] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6161.052861] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6169.063627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.074491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6169.080092] Lustre: Skipped 3 previous similar messages [ 6169.080939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6169.080943] Lustre: Skipped 4 previous similar messages [ 6169.114184] LustreError: Skipped 1 previous similar message [ 6173.755957] Lustre: server umount lustre-OST0000 complete [ 6183.745040] Lustre: server umount lustre-MDT0000 complete [ 6184.427429] LustreError: 179334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.441286] LustreError: 179334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6188.008325] LustreError: 179313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789584724 with bad export cookie 16088732836415741139 [ 6188.009888] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.023170] LustreError: 179313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6188.432083] Lustre: server umount lustre-MDT0001 complete [ 6200.109535] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6203.260692] Key type lgssc unregistered [ 6203.587027] LNet: 183155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6203.604820] LNetError: 183155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6203.623908] LNet: Removed LNI 192.168.204.152@tcp [ 6204.587126] Key type .llcrypt unregistered [ 6204.590632] Key type ._llcrypt unregistered [ 6233.145812] Key type ._llcrypt registered [ 6233.149312] Key type .llcrypt registered [ 6233.314464] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6234.873713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6234.958264] alg: No test for adler32 (adler32-zlib) [ 6236.275433] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6236.689626] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6238.424297] Key type lgssc registered [ 6239.898309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6250.542430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6250.581699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.268982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6257.903892] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6261.734982] LustreError: 184571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.766557] LustreError: 184571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6266.863155] LustreError: 184572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.190235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.874230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6274.007206] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6285.175734] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6293.182107] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6302.309368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6302.638662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6310.238785] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6319.263613] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6322.117950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6344.676879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.686203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.704396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6347.749516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6347.754593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6347.764731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.384553] Lustre: server umount lustre-OST0000 complete [ 6355.939448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6355.952840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.973716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6355.978944] Lustre: Skipped 1 previous similar message [ 6357.986581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.988471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.692185] Lustre: server umount lustre-MDT0000 complete [ 6363.107535] LustreError: 185300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.129486] LustreError: 185300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6363.734942] LustreError: 184552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789584900 with bad export cookie 17450323223132734607 [ 6363.739790] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6363.748169] LustreError: 184552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6364.132429] Lustre: server umount lustre-MDT0001 complete [ 6375.734490] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6378.617280] Key type lgssc unregistered [ 6378.887403] LNet: 188129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6378.900317] LNetError: 188129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6378.929291] LNet: Removed LNI 192.168.204.152@tcp [ 6379.690835] Key type .llcrypt unregistered [ 6379.692789] Key type ._llcrypt unregistered [ 6402.884733] Key type ._llcrypt registered [ 6402.890122] Key type .llcrypt registered [ 6403.079922] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6404.133866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6404.159801] alg: No test for adler32 (adler32-zlib) [ 6405.316753] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6405.582872] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6407.265604] Key type lgssc registered [ 6408.693140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6417.290894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6417.340475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.975856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.989473] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6429.155766] LustreError: 189542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.183546] LustreError: 189542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6433.251500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.712349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6438.267914] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6446.391767] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6452.023446] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6459.822364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6460.316572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.457935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6466.981729] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6475.462378] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.918392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.927593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6482.927674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6483.822206] Lustre: lustre-OST0000: Not available for connect from 192.168.204.52@tcp (stopping) [ 6483.835075] Lustre: Skipped 1 previous similar message [ 6488.037129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6488.047376] Lustre: Skipped 1 previous similar message [ 6488.769518] Lustre: server umount lustre-OST0000 complete [ 6497.472112] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff949ae02e4000.ost_server_uuid 50 [ 6499.411861] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6512.746609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6512.754414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6512.761392] Lustre: Skipped 2 previous similar messages [ 6519.097879] Lustre: server umount lustre-MDT0000 complete [ 6519.265413] LustreError: 189543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.293830] LustreError: 189543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6523.150834] LustreError: 189522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585059 with bad export cookie 14543139496447656783 [ 6523.155398] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.163388] LustreError: 189522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6523.392557] Lustre: server umount lustre-MDT0001 complete [ 6532.676878] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:57:47 (1789585067) [ 6540.066410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.531479] LustreError: 193459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.621736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.315605] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6545.892382] LustreError: 193460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.987881] LustreError: 193459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6554.273628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.959211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6560.376178] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6569.055531] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6575.890921] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6582.982212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6583.459635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.531480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6590.582184] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6598.763872] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6606.951319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.228397] Lustre: lustre-OST0001: new disk, initializing [ 6607.232911] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6607.239483] Lustre: 196302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6607.351887] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.117738] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6616.700199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6616.710840] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6616.852543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6627.162607] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.883466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6630.895057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.929429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6630.938933] Lustre: Skipped 3 previous similar messages [ 6631.913664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6636.640684] Lustre: server umount lustre-OST0000 complete [ 6637.549177] LustreError: 195283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.575935] LustreError: 195283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6644.388807] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6644.653417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6650.390986] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6650.657734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6653.754916] LustreError: 195285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.779362] LustreError: 195285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 6667.033320] Lustre: server umount lustre-OST0001 complete [ 6672.865061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6672.872648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.888682] Lustre: Skipped 1 previous similar message [ 6672.896947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.902353] Lustre: Skipped 4 previous similar messages [ 6681.685243] Lustre: server umount lustre-MDT0000 complete [ 6684.641309] Lustre: 188852:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789585204/real 1789585204] req@000000007e80b281 x1876515819189888/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789585220 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6684.672804] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.690356] Lustre: Skipped 1 previous similar message [ 6685.757491] LustreError: 193441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585222 with bad export cookie 14543139496447657854 [ 6685.757905] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.778489] LustreError: 193441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6686.126948] Lustre: server umount lustre-MDT0001 complete [ 6713.257902] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:00:48 (1789585248) [ 6720.258728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.553972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6720.609996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6720.871922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.764909] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6736.514561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.702091] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6736.776865] Lustre: 199743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6742.222783] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6750.719275] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.389097] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6765.248037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.509532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6772.834458] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6772.858389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6781.309089] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.897671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.016214] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6789.220047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.225631] Lustre: Skipped 2 previous similar messages [ 6796.072348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6797.310968] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6807.202492] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6811.633167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.639263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.645098] Lustre: Skipped 1 previous similar message [ 6811.649387] Lustre: Skipped 4 previous similar messages [ 6816.600858] Lustre: server umount lustre-OST0000 complete [ 6816.751104] LustreError: 201606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.773370] LustreError: 201606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6837.223424] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6837.226390] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.234237] LustreError: Skipped 1 previous similar message [ 6837.259755] Lustre: Skipped 1 previous similar message [ 6838.717731] Lustre: server umount lustre-OST0001 complete [ 6844.904191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6844.910487] Lustre: Skipped 3 previous similar messages [ 6850.118628] Lustre: server umount lustre-MDT0000 complete [ 6854.922857] LustreError: 199734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585391 with bad export cookie 14543139496447659261 [ 6854.927165] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.934901] LustreError: 199734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6855.432151] Lustre: server umount lustre-MDT0001 complete [ 6883.957476] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:03:38 (1789585418) [ 6903.014390] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 6912.497346] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6951.362716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.645473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6951.684902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6951.779402] Lustre: lustre-MDT0000: new disk, initializing [ 6951.963149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.987090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6956.887232] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6968.931711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.100738] Lustre: 207855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6969.176707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6969.180439] Lustre: Skipped 1 previous similar message [ 6969.298756] Lustre: lustre-MDT0001: new disk, initializing [ 6969.440533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6969.458218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6974.417898] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6985.722482] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6992.685604] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7000.975706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.221755] Lustre: lustre-OST0000: new disk, initializing [ 7001.227616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7001.234302] Lustre: 209822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7008.908247] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7011.409257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7011.425161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7011.552174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7019.308918] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.413508] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7024.627612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7029.907746] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7030.147227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7032.290381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7032.292205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.298281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7032.298289] Lustre: Skipped 2 previous similar messages [ 7032.299681] LustreError: Skipped 2 previous similar messages [ 7032.331503] Lustre: Skipped 3 previous similar messages [ 7038.295515] Lustre: server umount lustre-OST0000 complete [ 7048.661873] Lustre: server umount lustre-MDT0000 complete [ 7052.425593] LustreError: 207848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585588 with bad export cookie 14543139496447661046 [ 7052.427043] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7052.453059] LustreError: 207848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7052.828851] Lustre: server umount lustre-MDT0001 complete [ 7062.672430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.226116] LustreError: 212041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.250903] LustreError: 212041:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 7067.775563] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7076.770823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.716781] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7089.484709] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7094.930733] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7103.163075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7112.117289] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7125.523578] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7133.025157] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7133.305461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7138.352722] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7138.617502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7142.886899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.904460] Lustre: Skipped 2 previous similar messages [ 7142.909427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7146.847503] Lustre: server umount lustre-OST0000 complete [ 7153.787382] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7158.611911] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7189.639719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.054942] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7206.108706] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.314274] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7208.434706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7208.450686] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7208.451083] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7208.477912] Lustre: Skipped 1 previous similar message [ 7213.540744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7213.568232] LustreError: Skipped 1 previous similar message [ 7213.568769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.589880] Lustre: Skipped 9 previous similar messages [ 7218.840749] Lustre: server umount lustre-OST0000 complete [ 7229.421154] Lustre: server umount lustre-MDT0000 complete [ 7233.279096] LustreError: 215156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585769 with bad export cookie 14543139496447661928 [ 7233.279730] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.285293] LustreError: 215156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7233.553384] Lustre: server umount lustre-MDT0001 complete [ 7242.900842] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:09:37 (1789585777) [ 7250.244374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.809162] LustreError: 218013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.845640] LustreError: 218013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7250.917086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.924555] Lustre: Skipped 6 previous similar messages [ 7255.691469] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7265.608665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7271.217886] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7279.337143] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7285.522384] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7292.432443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7299.998961] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7308.153238] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7313.270212] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7313.523785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7318.395335] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7318.640943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7325.715663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.973881] Lustre: lustre-OST0001: new disk, initializing [ 7325.978516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7325.985644] Lustre: 221384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7333.142110] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7334.057506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7334.063684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7334.165603] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7342.743923] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7347.333306] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7347.649103] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7352.812935] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7353.062355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7359.973112] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7359.980891] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.981052] LustreError: Skipped 1 previous similar message [ 7359.994093] Lustre: Skipped 5 previous similar messages [ 7361.286271] Lustre: server umount lustre-OST0001 complete [ 7367.315611] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7367.560502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7371.218257] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7371.428135] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7400.363754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.078482] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7406.091677] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7406.093145] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7406.112130] Lustre: Skipped 1 previous similar message [ 7406.223987] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7414.124685] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7416.056922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7425.530242] Lustre: server umount lustre-OST0001 complete [ 7448.640520] Lustre: server umount lustre-OST0000 complete [ 7459.052340] Lustre: server umount lustre-MDT0000 complete [ 7462.807475] LustreError: 217995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585999 with bad export cookie 14543139496447662894 [ 7462.829188] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7477.216126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7477.635905] Lustre: server umount lustre-MDT0001 complete [ 7500.244924] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:13:55 (1789586035) [ 7506.410873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.795669] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7506.830750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7511.827268] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7519.738665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.893136] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7519.925321] Lustre: 226631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7526.269725] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7534.836686] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7540.989714] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7549.386215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7549.599547] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7556.752620] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7564.734064] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.779660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7573.844693] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7580.464965] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7588.161851] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7589.514694] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7594.171704] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7594.496811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7600.625727] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7600.944533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7608.348884] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff949ad214c800.ost_server_uuid 50 [ 7610.445667] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff949ad214c800.ost_server_uuid in IDLE state after 0 sec [ 7611.526582] Lustre: Permanently deactivating lustre-OST0001 [ 7611.542579] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7614.782341] Lustre: Permanently reactivating lustre-OST0001 [ 7618.532407] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7618.540771] LustreError: Skipped 4 previous similar messages [ 7618.543101] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.549798] Lustre: Skipped 7 previous similar messages [ 7618.552158] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7618.558449] Lustre: Skipped 17 previous similar messages [ 7624.747217] Lustre: server umount lustre-OST0001 complete [ 7629.286701] LustreError: 230260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.318133] LustreError: 230260:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 35 previous similar messages [ 7635.355939] Lustre: server umount lustre-OST0000 complete [ 7645.601250] Lustre: server umount lustre-MDT0000 complete [ 7649.250431] LustreError: 226623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586185 with bad export cookie 14543139496447664175 [ 7649.259265] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7649.259415] LustreError: 226623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7649.608895] Lustre: server umount lustre-MDT0001 complete [ 7669.142772] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:16:43 (1789586203) [ 7678.258747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.517520] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7678.572404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7678.583921] Lustre: Skipped 5 previous similar messages [ 7683.427515] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7692.711557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.829751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7692.854214] Lustre: 233331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7697.614647] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7704.629414] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7710.478796] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7717.480247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.698862] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7717.725269] Lustre: Permanently deactivating lustre-OST0000 [ 7717.754708] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7723.039045] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7723.901484] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7730.997345] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7737.241285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7737.311922] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7743.270530] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7744.172709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7750.542945] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7752.393076] Lustre: Permanently reactivating lustre-OST0000 [ 7752.419030] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7752.424370] Lustre: Skipped 2 previous similar messages [ 7761.058576] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7761.060355] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 000000008e184049, cur 1789586297 deadline 1789586289 last 1789586259 [ 7761.072466] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7761.097667] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7761.178527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7776.471377] Lustre: server umount lustre-OST0001 complete [ 7795.168274] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7795.271571] Lustre: server umount lustre-OST0000 complete [ 7805.440088] Lustre: server umount lustre-MDT0000 complete [ 7808.800695] LustreError: 233322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586345 with bad export cookie 14543139496447665855 [ 7808.812650] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.819748] LustreError: 233322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7809.080574] Lustre: server umount lustre-MDT0001 complete [ 7818.272596] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7821.555372] Key type lgssc unregistered [ 7821.983759] LNet: 238211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7821.991995] LNetError: 238211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7822.029248] LNet: Removed LNI 192.168.204.152@tcp [ 7823.201180] Key type .llcrypt unregistered [ 7823.203846] Key type ._llcrypt unregistered [ 7852.387604] Key type ._llcrypt registered [ 7852.401385] Key type .llcrypt registered [ 7852.607671] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7853.187795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7853.302784] alg: No test for adler32 (adler32-zlib) [ 7854.360456] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7854.631272] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7856.312925] Key type lgssc registered [ 7857.453543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7866.782239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7866.804856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.280421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7872.572179] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7878.629329] LustreError: 239674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.644077] LustreError: 239674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7881.463323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7881.561355] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7881.588929] Lustre: 239663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7881.595770] Lustre: Permanently deactivating lustre-MDT0001 [ 7881.605461] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7881.871189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7886.212095] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7887.913938] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7893.274576] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7898.190879] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7904.777129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7905.073690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7907.119233] LustreError: 241517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.165123] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7911.857309] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7912.418567] LustreError: 241516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.492716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7918.561728] LustreError: 241517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.595151] LustreError: 241517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7920.057755] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7923.686318] LustreError: 241518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.717614] LustreError: 241518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7927.144639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.332676] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7932.409197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7932.437078] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7933.959277] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7942.181619] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7946.128821] Lustre: Permanently reactivating lustre-MDT0001 [ 7946.145291] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7946.149911] Lustre: Skipped 2 previous similar messages [ 7949.875904] Lustre: lustre-MDT0001: Client 61bf8a1c-b5de-4a6d-b1ed-59cc4520d27c (at 192.168.204.52@tcp) reconnecting [ 7953.514263] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.530990] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7953.539451] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7953.554329] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7957.136937] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7958.732710] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7964.812178] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7965.082187] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7970.278173] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7970.500276] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7973.202991] Lustre: Permanently deactivating lustre-MDT0001 [ 7973.215152] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7973.225714] Lustre: Skipped 2 previous similar messages [ 7981.177447] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7994.337714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7994.345393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.360706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7994.863751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7994.869353] Lustre: Skipped 2 previous similar messages [ 7997.281852] Lustre: server umount lustre-MDT0000 complete [ 7999.977515] LustreError: 239675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7999.999698] LustreError: 239675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8001.138294] LustreError: 243224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586537 with bad export cookie 10188713708416649124 [ 8001.147023] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8001.158638] LustreError: 243224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8001.539421] Lustre: server umount lustre-MDT0001 complete [ 8007.788088] Lustre: server umount lustre-OST0000 complete [ 8021.175846] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:22:35 (1789586555) [ 8033.410428] Lustre: server umount lustre-OST0001 complete [ 8041.998988] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 8051.759171] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8091.589196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.952431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8091.962233] Lustre: Skipped 2 previous similar messages [ 8091.986531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8092.085285] Lustre: lustre-MDT0000: new disk, initializing [ 8092.206467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.227841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8097.720136] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8109.755195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.915963] Lustre: 247961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.972296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8109.976268] Lustre: Skipped 1 previous similar message [ 8110.061177] Lustre: lustre-MDT0001: new disk, initializing [ 8110.166149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8110.211042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8110.225415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8115.877608] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8127.695298] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8134.723312] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8143.807394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8144.267880] Lustre: lustre-OST0000: new disk, initializing [ 8144.270576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8144.278326] Lustre: 249929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8144.436732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.760936] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8154.696649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8154.708799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8154.841236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8163.405731] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.008598] LustreError: 247966:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8175.983208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.032105] LustreError: 247966:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8176.088072] LustreError: 247968:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8176.306649] Lustre: lustre-OST0001: new disk, initializing [ 8176.316369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8176.331942] Lustre: 251043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8176.465682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8184.000565] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8185.499053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8185.527513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8185.679701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8186.136916] LustreError: 247968:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8195.775265] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8197.619459] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8201.191178] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.208333] Lustre: Skipped 3 previous similar messages [ 8201.211812] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8201.211969] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8201.231161] LustreError: Skipped 1 previous similar message [ 8206.056935] Lustre: server umount lustre-OST0001 complete [ 8206.313897] LustreError: 250392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8206.343713] LustreError: 250392:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8226.791162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8226.793485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8226.798912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8226.798921] Lustre: Skipped 1 previous similar message [ 8226.806823] LustreError: Skipped 1 previous similar message [ 8226.831129] Lustre: Skipped 1 previous similar message [ 8229.689503] Lustre: server umount lustre-OST0000 complete [ 8237.026925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.032712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.046563] Lustre: Skipped 1 previous similar message [ 8237.061417] Lustre: Skipped 1 previous similar message [ 8240.650473] Lustre: server umount lustre-MDT0000 complete [ 8242.146366] LustreError: 247972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8242.160338] LustreError: 247972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8244.854509] LustreError: 247952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586781 with bad export cookie 10188713708416651938 [ 8244.857873] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8244.867547] LustreError: 247952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8251.667910] Lustre: server umount lustre-MDT0001 complete [ 8264.626245] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8268.313868] Key type lgssc unregistered [ 8268.686219] LNet: 253078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8268.698777] LNetError: 253078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8268.721907] LNet: Removed LNI 192.168.204.152@tcp [ 8269.953379] Key type .llcrypt unregistered [ 8269.955192] Key type ._llcrypt unregistered [ 8317.104293] Key type ._llcrypt registered [ 8317.110198] Key type .llcrypt registered [ 8317.304902] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8318.935779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8319.004920] alg: No test for adler32 (adler32-zlib) [ 8320.253762] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8320.596735] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 8322.264277] Key type lgssc registered [ 8323.569568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8332.492596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8332.515724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.883909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8333.933238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8334.205197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.935898] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8348.599951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.771240] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8348.838693] Lustre: 255286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8349.283820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8354.364221] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8362.574967] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8369.945982] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8377.263365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.572753] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8377.882072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8384.996630] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8387.837596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8394.520727] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8449.509927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8449.511590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.518573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8449.522500] LustreError: Skipped 1 previous similar message [ 8449.572161] Lustre: Skipped 1 previous similar message [ 8453.341637] Lustre: server umount lustre-OST0000 complete [ 8458.421943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8476.060626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8476.549516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.485770] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8488.827210] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8488.857282] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8489.895038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8489.934207] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8494.909391] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8536.049051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8536.066081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.067808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.067827] Lustre: Skipped 1 previous similar message [ 8541.159545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8541.168606] Lustre: Skipped 1 previous similar message [ 8541.721924] Lustre: server umount lustre-OST0000 complete [ 8548.834137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8548.845269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.868872] Lustre: Skipped 1 previous similar message [ 8548.883828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.899556] Lustre: Skipped 1 previous similar message [ 8551.421692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8552.992447] Lustre: server umount lustre-MDT0000 complete [ 8557.421598] LustreError: 255277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587093 with bad export cookie 10256592155894978026 [ 8557.424026] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8557.444522] LustreError: 255277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8557.710594] Lustre: server umount lustre-MDT0001 complete [ 8571.899599] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8574.922455] Key type lgssc unregistered [ 8575.260166] LNet: 260714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8575.272384] LNetError: 260714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8575.323104] LNet: Removed LNI 192.168.204.152@tcp [ 8576.582907] Key type .llcrypt unregistered [ 8576.585982] Key type ._llcrypt unregistered [ 8604.242275] Key type ._llcrypt registered [ 8604.247786] Key type .llcrypt registered [ 8604.365839] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8605.460936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8605.906130] alg: No test for adler32 (adler32-zlib) [ 8607.089156] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8607.347672] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 8609.096196] Key type lgssc registered [ 8610.435457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8618.680627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8618.707168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8620.290757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.625140] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8629.735718] LustreError: 262129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.765150] LustreError: 262129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8634.850991] LustreError: 262130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.133043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.524448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8640.339154] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8649.641310] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8656.967812] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8665.667656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8666.117803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8673.364403] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8683.430386] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8689.662760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8721.379914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8721.389555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8721.424939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8725.480151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8725.486917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8725.499152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.036460] Lustre: server umount lustre-OST0000 complete [ 8732.641283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8732.663433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8732.692742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8732.705305] Lustre: Skipped 1 previous similar message [ 8735.715150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8735.717305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.149052] Lustre: server umount lustre-MDT0000 complete [ 8740.865393] LustreError: 262858:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.898344] LustreError: 262858:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8742.594978] LustreError: 262110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587278 with bad export cookie 14105735067392404631 [ 8742.607476] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8742.612664] LustreError: 262110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8743.060701] Lustre: server umount lustre-MDT0001 complete [ 8756.202132] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8759.934227] Key type lgssc unregistered [ 8760.329945] LNet: 266461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8760.338512] LNetError: 266461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8760.350603] LNet: Removed LNI 192.168.204.152@tcp [ 8761.227432] Key type .llcrypt unregistered [ 8761.229658] Key type ._llcrypt unregistered [ 8780.032902] Key type ._llcrypt registered [ 8780.035466] Key type .llcrypt registered [ 8780.146709] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8781.907120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8781.928370] alg: No test for adler32 (adler32-zlib) [ 8783.006535] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8783.279525] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 8785.056216] Key type lgssc registered [ 8786.483165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8795.659646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8795.684416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.228143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8802.101231] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8807.418451] LustreError: 267679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.470623] LustreError: 267679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8812.282187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.519829] LustreError: 267680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8812.872455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8819.749179] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8831.245932] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8837.414779] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8845.142802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8845.553130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8853.276809] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8862.146853] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8867.069736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8880.995928] Lustre: server umount lustre-OST0000 complete [ 8887.777518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.779074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.805404] Lustre: Skipped 1 previous similar message [ 8892.898256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8892.915558] Lustre: Skipped 2 previous similar messages [ 8895.055118] Lustre: server umount lustre-MDT0000 complete [ 8898.023846] LustreError: 267679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.038381] LustreError: 267679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8898.409583] LustreError: 270320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587434 with bad export cookie 8754317254195577986 [ 8898.413921] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8898.421751] LustreError: 270320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8907.984833] Lustre: server umount lustre-MDT0001 complete [ 8920.348977] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8923.845275] Key type lgssc unregistered [ 8924.230342] LNet: 271360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8924.239172] LNetError: 271360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8924.262381] LNet: Removed LNI 192.168.204.152@tcp [ 8925.663115] Key type .llcrypt unregistered [ 8925.672981] Key type ._llcrypt unregistered [ 8950.665656] Key type ._llcrypt registered [ 8950.667311] Key type .llcrypt registered [ 8950.807743] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8952.240963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8952.303296] alg: No test for adler32 (adler32-zlib) [ 8953.363444] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8953.570843] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 8955.200209] Key type lgssc registered [ 8956.602366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8965.836554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8965.891321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.536833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8973.302740] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 8976.866585] LustreError: 272773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.896865] LustreError: 272773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8981.991051] LustreError: 272774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8984.752826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.313151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.552372] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9000.158783] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.653526] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9013.431340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.871309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.442464] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9030.636126] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9037.428136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9066.472281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9066.479195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9066.493519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9068.518042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9068.518695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9068.523030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.528286] Lustre: Skipped 1 previous similar message [ 9070.039278] Lustre: server umount lustre-OST0000 complete [ 9077.740989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9077.754379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9077.778748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9080.415046] Lustre: server umount lustre-MDT0000 complete [ 9083.889873] LustreError: 272773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.909804] LustreError: 272773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9084.856881] LustreError: 272754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587621 with bad export cookie 11327883971591696405 [ 9084.867396] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9084.877322] LustreError: 272754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9085.461578] Lustre: server umount lustre-MDT0001 complete [ 9097.900931] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 9100.972542] Key type lgssc unregistered [ 9101.260922] LNet: 277102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9101.277031] LNetError: 277102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9101.303718] LNet: Removed LNI 192.168.204.152@tcp [ 9102.267966] Key type .llcrypt unregistered [ 9102.273140] Key type ._llcrypt unregistered [ 9120.670387] Key type ._llcrypt registered [ 9120.673823] Key type .llcrypt registered [ 9120.821379] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9122.231595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9122.325924] alg: No test for adler32 (adler32-zlib) [ 9123.505323] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 9123.799289] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 9125.528228] Key type lgssc registered [ 9127.044213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9136.760061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9136.785132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9138.167502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.966902] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9148.389248] LustreError: 278336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.431648] LustreError: 278336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9153.506544] LustreError: 278338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9153.745772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.993338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9160.521040] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9172.021860] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9178.884035] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9185.321611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.599198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.563926] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9202.331851] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9209.154642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9224.678425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.683501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9224.712858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.696492] Lustre: server umount lustre-OST0000 complete [ 9230.818473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9230.831955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.848136] Lustre: Skipped 1 previous similar message [ 9230.857446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9230.865669] Lustre: Skipped 1 previous similar message [ 9234.916186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9234.920734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.969927] Lustre: server umount lustre-MDT0000 complete [ 9240.047216] LustreError: 278334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.076403] LustreError: 278334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9241.183075] LustreError: 278302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587777 with bad export cookie 13088908136918604121 [ 9241.194533] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9241.211353] LustreError: 278302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9241.651524] Lustre: server umount lustre-MDT0001 complete [ 9252.999493] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 9256.246251] Key type lgssc unregistered [ 9256.669854] LNet: 282057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9256.680722] LNetError: 282057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9256.713348] LNet: Removed LNI 192.168.204.152@tcp [ 9257.661680] Key type .llcrypt unregistered [ 9257.665649] Key type ._llcrypt unregistered [ 9271.167843] Key type ._llcrypt registered [ 9271.169757] Key type .llcrypt registered [ 9271.343340] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9290.242433] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 9309.641277] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9310.770727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9310.804473] alg: No test for adler32 (adler32-zlib) [ 9312.156715] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 9312.722372] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 9314.536743] Key type lgssc registered [ 9316.271840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9358.228435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9358.333570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9359.665087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9359.704866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9359.777593] Lustre: lustre-MDT0000: new disk, initializing [ 9359.903635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9359.919677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9365.758832] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9379.013761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.212354] Lustre: 285611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9379.298354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9379.305940] Lustre: Skipped 1 previous similar message [ 9379.437539] Lustre: lustre-MDT0001: new disk, initializing [ 9379.553435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9379.616599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9379.621397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9384.231243] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9396.463530] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9403.733224] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9411.144117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9411.349330] Lustre: lustre-OST0000: new disk, initializing [ 9411.356196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9411.363714] Lustre: 287580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9411.497188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.785682] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9420.363724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9420.376140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9420.514318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9430.131879] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9436.517526] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9436.812305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9441.988729] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9442.268160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9446.371414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9446.372833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.383506] LustreError: Skipped 1 previous similar message [ 9446.386465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9446.409584] Lustre: Skipped 1 previous similar message [ 9446.436973] Lustre: Skipped 1 previous similar message [ 9450.870694] Lustre: server umount lustre-OST0000 complete [ 9456.621928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.639368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9456.652018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9456.660148] Lustre: Skipped 1 previous similar message [ 9461.488820] Lustre: server umount lustre-MDT0000 complete [ 9465.642452] LustreError: 285603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588002 with bad export cookie 8273857375106616878 [ 9465.648619] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.664910] LustreError: 285603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9466.071978] Lustre: server umount lustre-MDT0001 complete [ 9475.207900] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:46:50 (1789588010) [ 9482.832440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.302346] LustreError: 289992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9483.320420] LustreError: 289992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9483.409675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.868938] LustreError: 289993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9489.099527] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9492.963120] LustreError: 289992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9498.084807] LustreError: 289993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9500.746423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.195333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9507.179356] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9516.361095] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9527.299419] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9538.213094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.734048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.862736] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9554.504468] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9566.689880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9566.699548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.729273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.731664] Lustre: Skipped 1 previous similar message [ 9568.740627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9568.743900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9568.750032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9570.440783] Lustre: server umount lustre-OST0000 complete [ 9577.962497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9577.974575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.999446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9578.013722] Lustre: Skipped 1 previous similar message [ 9589.216283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9589.219872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.233805] Lustre: Skipped 5 previous similar messages [ 9589.531481] Lustre: server umount lustre-MDT0000 complete [ 9594.215583] LustreError: 289974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588130 with bad export cookie 8273857375106617767 [ 9594.228981] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9594.234600] LustreError: 289974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9594.691777] Lustre: server umount lustre-MDT0001 complete [ 9607.197755] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 9610.114622] Key type lgssc unregistered [ 9610.501812] LNet: 293529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9610.510954] LNetError: 293529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9610.530688] LNet: Removed LNI 192.168.204.152@tcp [ 9611.685936] Key type .llcrypt unregistered [ 9611.688118] Key type ._llcrypt unregistered [ 9650.488344] Key type ._llcrypt registered [ 9650.491308] Key type .llcrypt registered [ 9650.663725] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 9668.155450] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9669.939159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9670.095063] alg: No test for adler32 (adler32-zlib) [ 9671.352711] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 9671.825241] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 9673.689430] Key type lgssc registered [ 9675.250989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9728.987736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9729.014986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.455260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9730.514916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9730.721743] Lustre: lustre-MDT0000: new disk, initializing [ 9730.870640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.906410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9736.498076] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9748.370250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.536064] Lustre: 297358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9748.585162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9748.591639] Lustre: Skipped 1 previous similar message [ 9748.685078] Lustre: lustre-MDT0001: new disk, initializing [ 9748.851367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9748.896389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9748.904215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9754.068159] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9768.088352] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9776.104353] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9784.478324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9784.814663] Lustre: lustre-OST2710: new disk, initializing [ 9784.823348] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9784.837776] Lustre: 299326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9784.966902] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9790.529090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9790.537126] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9790.642225] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9792.617347] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9803.081303] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9811.686815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.945396] Lustre: lustre-OST03e8: new disk, initializing [ 9811.950159] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9811.968502] Lustre: 300393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9812.094726] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9817.206328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9817.230697] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9817.325748] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9819.664923] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 9829.358092] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9838.486693] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9838.659221] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9844.223388] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9844.579937] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9849.301366] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9849.517791] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9856.722545] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9857.116536] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9863.654549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9863.660552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9863.688338] Lustre: Skipped 2 previous similar messages [ 9863.705647] Lustre: Skipped 3 previous similar messages [ 9868.773343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9869.743165] Lustre: server umount lustre-MDT0000 complete [ 9874.622316] LustreError: 297349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588410 with bad export cookie 7656432565738067118 [ 9874.625744] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9874.647921] LustreError: 297349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9875.143805] Lustre: server umount lustre-MDT0001 complete [ 9890.440809] Lustre: server umount lustre-OST2710 complete [ 9905.474747] Lustre: server umount lustre-OST03e8 complete [ 9933.029502] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 9943.803713] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 9981.993565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9982.311644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9982.359280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9982.471256] Lustre: lustre-MDT0000: new disk, initializing [ 9982.548273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9982.572713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9987.417824] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10000.380851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10000.632314] Lustre: 305649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10000.732409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10000.735305] Lustre: Skipped 1 previous similar message [10000.818734] Lustre: lustre-MDT0001: new disk, initializing [10000.995628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10001.073067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10001.098378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10007.512326] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10021.535734] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10028.876718] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10037.543839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10037.721757] Lustre: lustre-OST0000: new disk, initializing [10037.724748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10037.729596] Lustre: 307623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10037.813607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10044.472366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10044.481544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10044.625897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10045.262456] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10056.130526] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10061.934743] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10062.129354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10067.947247] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10068.335346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10075.617323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10075.631377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.646090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10075.661405] Lustre: Skipped 2 previous similar messages [10075.673224] Lustre: Skipped 4 previous similar messages [10076.868744] Lustre: server umount lustre-OST0000 complete [10080.738545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10080.740426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10080.746585] Lustre: Skipped 1 previous similar message [10085.862083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10085.868233] Lustre: Skipped 2 previous similar messages [10086.685809] Lustre: server umount lustre-MDT0000 complete [10090.438057] LustreError: 305643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588626 with bad export cookie 7656432565738068812 [10090.443309] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10090.453273] LustreError: 305643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10090.757345] Lustre: server umount lustre-MDT0001 complete [10101.882779] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:57:16 (1789588636) [10103.897236] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10105.945466] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:57:20 (1789588640) [10110.219216] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10145.927098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.128532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10146.141885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10146.281504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10150.748817] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10160.742759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.962421] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10160.998198] Lustre: 311484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10167.117570] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10190.069907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.318566] Lustre: Denying initial registration attempt from nid 192.168.204.152@tcp, specified as failover [10190.326907] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10190.335805] LustreError: 312937:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [10190.345896] LustreError: 312937:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10190.358926] LustreError: 312937:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10190.450536] Lustre: server umount lustre-OST0000 complete [10190.454637] LustreError: 312937:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10202.084186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10202.089049] LustreError: Skipped 1 previous similar message [10202.094485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10202.115874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10212.323598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10212.628915] Lustre: server umount lustre-MDT0000 complete [10212.837106] LustreError: 311495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.858352] LustreError: 311495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10217.345114] LustreError: 311474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588753 with bad export cookie 7656432565738069687 [10217.348167] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.357739] LustreError: 311474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10217.910775] Lustre: server umount lustre-MDT0001 complete [10230.616835] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [10234.083435] Key type lgssc unregistered [10234.435916] LNet: 314058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10234.452414] LNetError: 314058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10234.483814] LNet: Removed LNI 192.168.204.152@tcp [10235.760362] Key type .llcrypt unregistered [10235.767129] Key type ._llcrypt unregistered [10250.160338] Key type ._llcrypt registered [10250.163340] Key type .llcrypt registered [10250.291455] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10251.989771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10252.064360] alg: No test for adler32 (adler32-zlib) [10253.398957] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [10253.811710] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [10255.528823] Key type lgssc registered [10256.839068] Lustre: Echo OBD driver; http://www.lustre.org/ [10312.126230] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10322.784527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10322.807979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.060859] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10324.117872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10324.348662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10329.919735] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10340.432592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10340.563960] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10340.620915] Lustre: 316821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10341.054516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10346.242154] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10354.815326] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10361.806264] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10382.440684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10382.636895] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10382.860582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10389.250859] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10399.478969] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10403.299611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10403.308687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.327757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10403.810891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.830834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10408.323053] Lustre: server umount lustre-OST0000 complete [10414.049715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.073826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10414.083333] Lustre: Skipped 1 previous similar message [10419.170627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.191536] Lustre: Skipped 1 previous similar message [10424.295679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10424.310912] Lustre: Skipped 1 previous similar message [10427.360117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10427.563409] Lustre: server umount lustre-MDT0000 complete [10431.723691] LustreError: 316815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588968 with bad export cookie 9167247654978416904 [10431.730183] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10431.740966] LustreError: 316815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10432.148698] Lustre: server umount lustre-MDT0001 complete [10444.321380] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [10447.675506] Key type lgssc unregistered [10448.115265] LNet: 320409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10448.130726] LNetError: 320409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10449.194480] LNet: Removed LNI 192.168.204.152@tcp [10450.243343] Key type .llcrypt unregistered [10450.246780] Key type ._llcrypt unregistered [10477.511350] Key type ._llcrypt registered [10477.515273] Key type .llcrypt registered [10477.694741] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10479.764539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10479.850579] alg: No test for adler32 (adler32-zlib) [10481.152763] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [10481.554575] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [10483.312326] Key type lgssc registered [10484.644082] Lustre: Echo OBD driver; http://www.lustre.org/ [10495.494614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10495.510474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.911252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10502.063606] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10507.242361] LustreError: 321826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.258835] LustreError: 321826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10512.026900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.353282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10512.539121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10518.340966] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10527.907814] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10536.031668] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10546.044905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.576372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10554.645995] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10564.816325] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10582.346901] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [10582.504635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.516899] Lustre: Skipped 2 previous similar messages [10583.792615] Lustre: server umount lustre-MDT0000 complete [10587.447546] LustreError: 321821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.285732] LustreError: 321805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589124 with bad export cookie 15422660356652292919 [10588.303395] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.313919] LustreError: 321805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10592.557404] LustreError: 324085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10592.579915] LustreError: 324085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10592.596392] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.52@tcp (stopping) [10592.603346] Lustre: Skipped 3 previous similar messages [10592.738543] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.085954] Lustre: server umount lustre-MDT0001 complete [10602.667484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10612.961432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.223437] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd6084be1b1c2786b [10613.248714] Lustre: MGC192.168.204.152@tcp: Connection restored to 0@lo (at 0@lo) [10613.777492] LustreError: 323657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.797056] LustreError: 323657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10613.889267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10613.957513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10617.954815] LustreError: 323658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10617.981661] LustreError: 323658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10619.354368] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10623.101385] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10628.608183] LustreError: 325442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.646350] LustreError: 325442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10631.238457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.590759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10636.809675] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10638.456367] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10646.568863] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10648.194816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10654.846585] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10656.784954] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10664.936549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10664.952753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10664.983102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10664.999780] Lustre: Skipped 2 previous similar messages [10667.490933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10667.505716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.895782] Lustre: server umount lustre-OST0000 complete [10677.737133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10677.744729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10677.758886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10677.781125] Lustre: Skipped 1 previous similar message [10677.804476] Lustre: Skipped 3 previous similar messages [10682.849975] LustreError: 325446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.011373] Lustre: server umount lustre-MDT0000 complete [10687.542143] LustreError: 321806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589223 with bad export cookie 15422660356652300395 [10687.546672] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10687.553288] LustreError: 321806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10687.820863] Lustre: server umount lustre-MDT0001 complete [10702.559816] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [10705.457638] Key type lgssc unregistered [10705.836792] LNet: 328117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10705.846130] LNetError: 328117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10705.873952] LNet: Removed LNI 192.168.204.152@tcp [10706.875377] Key type .llcrypt unregistered [10706.878466] Key type ._llcrypt unregistered [10732.051306] Key type ._llcrypt registered [10732.052775] Key type .llcrypt registered [10732.213971] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [10733.392391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10733.635117] alg: No test for adler32 (adler32-zlib) [10734.803751] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [10735.146453] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [10736.880800] Key type lgssc registered [10738.090179] Lustre: Echo OBD driver; http://www.lustre.org/ [10747.029668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10747.067069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.829135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10753.814624] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10759.141571] LustreError: 329532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10759.171669] LustreError: 329532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10763.859617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.292553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10769.934892] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10779.329269] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10786.345828] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10790.055717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10790.058887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10790.082134] Lustre: Skipped 1 previous similar message [10790.114930] Lustre: Skipped 1 previous similar message [10795.492225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10795.504359] Lustre: Skipped 1 previous similar message [10796.519108] Lustre: server umount lustre-MDT0000 complete [10800.609481] LustreError: 329533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10800.636270] LustreError: 329533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10802.634899] LustreError: 329514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589339 with bad export cookie 15627037859022638435 [10802.663517] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10803.152478] Lustre: server umount lustre-MDT0001 complete [10824.635841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10825.205854] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10825.255107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10825.655894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10831.908442] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10842.279634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.410262] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10842.458236] Lustre: 332121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10842.540595] LustreError: 332132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.719146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10848.233283] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10857.329901] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10865.307848] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10876.588800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10876.817504] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10876.841530] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10876.856639] LustreError: 334005:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10876.865175] LustreError: 334005:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10876.868162] LustreError: 334005:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10876.956787] Lustre: server umount lustre-OST0000 complete [10876.959148] LustreError: 334005:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10888.619637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10888.846381] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10889.186323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10894.516927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10895.577134] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10906.072739] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10916.399427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10916.553062] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10916.639526] Lustre: lustre-OST0001: new disk, initializing [10916.643704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10916.651879] Lustre: 335546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10916.771839] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10924.038864] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [10924.228697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10924.246076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10924.445751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10933.079847] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10939.882980] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.888276] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10939.891677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10939.905724] LustreError: Skipped 1 previous similar message [10939.913442] Lustre: Skipped 1 previous similar message [10942.174525] Lustre: server umount lustre-OST0001 complete [10945.009854] LustreError: 334472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.042090] LustreError: 334472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10950.122307] LustreError: 335919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.123979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10950.129897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10950.145406] LustreError: 335919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10950.158509] Lustre: Skipped 2 previous similar messages [10952.916653] Lustre: server umount lustre-OST0000 complete [10960.357587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10960.368190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.374515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10960.385390] Lustre: Skipped 1 previous similar message [10960.404165] Lustre: Skipped 2 previous similar messages [10964.035078] Lustre: server umount lustre-MDT0000 complete [10965.476578] LustreError: 332129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.494174] LustreError: 332129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10968.111443] LustreError: 332112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589504 with bad export cookie 15627037859022638981 [10968.122957] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10968.123324] LustreError: 332112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10968.637874] Lustre: server umount lustre-MDT0001 complete [10980.964815] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [10984.513512] Key type lgssc unregistered [10984.852420] LNet: 337510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10984.869553] LNetError: 337510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10984.886245] LNet: Removed LNI 192.168.204.152@tcp [10985.952720] Key type .llcrypt unregistered [10985.957119] Key type ._llcrypt unregistered [11067.317852] Key type ._llcrypt registered [11067.321846] Key type .llcrypt registered [11067.439576] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [11084.509299] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [11085.442412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11085.459581] alg: No test for adler32 (adler32-zlib) [11087.108996] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11087.509351] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [11089.296218] Key type lgssc registered [11091.538557] Lustre: Echo OBD driver; http://www.lustre.org/ [11138.741898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11138.813157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.251959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11140.283583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11140.397622] Lustre: lustre-MDT0000: new disk, initializing [11140.541733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11140.569551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11146.341457] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11158.828223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.033973] Lustre: 342458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11159.093567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11159.096609] Lustre: Skipped 1 previous similar message [11159.207909] Lustre: lustre-MDT0001: new disk, initializing [11159.411930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11159.503888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11159.527902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11164.766805] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11175.752811] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11182.145281] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11189.583612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11190.012547] Lustre: lustre-OST0000: new disk, initializing [11190.023572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11190.035650] Lustre: 344427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11190.174353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11197.066817] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11198.545643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11198.560814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11198.667423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11207.746493] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11212.365676] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11212.557179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11217.251181] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11217.604386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11224.570751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11224.571532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.589147] LustreError: Skipped 1 previous similar message [11224.613996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11224.627716] Lustre: Skipped 1 previous similar message [11225.817958] Lustre: server umount lustre-OST0000 complete [11229.665840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11229.673690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11229.682912] Lustre: Skipped 2 previous similar messages [11234.786996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11234.790266] Lustre: Skipped 2 previous similar messages [11235.867413] Lustre: server umount lustre-MDT0000 complete [11241.004536] LustreError: 342449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589777 with bad export cookie 11150427020529157055 [11241.018845] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11241.039262] LustreError: 342449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11242.026936] Lustre: server umount lustre-MDT0001 complete [11252.876126] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:16:27 (1789589787) [11262.449613] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:16:37 (1789589797) [11272.503794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.949183] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.982614] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11273.037636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11277.618672] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11278.312282] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.347832] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11283.425429] LustreError: 347184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.172472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.539696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11293.031379] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11304.030578] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11311.704869] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11319.818855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11320.226368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11328.485465] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11337.724857] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11350.321248] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11352.335698] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11354.368137] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11356.076671] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11358.555254] Lustre: Failing over lustre-MDT0000 [11358.890250] Lustre: server umount lustre-MDT0000 complete [11359.069214] LustreError: 349024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.224862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11360.232274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11363.302519] LustreError: 347184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11363.302938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11363.327065] LustreError: 347184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11363.364750] Lustre: Skipped 1 previous similar message [11373.537730] LustreError: 349024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.558862] LustreError: 349024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [11377.317535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.474125] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11377.806404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11377.888288] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11380.545069] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11382.804721] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11383.299676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11383.341289] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11383.409921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11394.098866] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11396.180143] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11400.002758] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11402.189308] Lustre: Failing over lustre-MDT0000 [11402.411369] Lustre: server umount lustre-MDT0000 complete [11403.753942] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.758443] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11403.771651] Lustre: Skipped 2 previous similar messages [11403.771990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11403.795182] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11420.141769] Lustre: 340678:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789589940/real 1789589940] req@00000000f21877b6 x1876520728619904/t0(0) o400->MGC192.168.204.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789589956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11420.169415] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11422.528833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.658715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11430.732218] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11431.743168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11435.458480] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11436.017499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11436.023273] Lustre: Skipped 2 previous similar messages [11436.067029] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11436.143740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11446.507479] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11448.407839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11456.482103] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11456.501949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11456.505136] Lustre: Skipped 1 previous similar message [11456.515673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11456.529082] Lustre: Skipped 2 previous similar messages [11460.531594] Lustre: server umount lustre-MDT0000 complete [11461.605795] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.620226] LustreError: 347183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [11464.716881] LustreError: 347165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590001 with bad export cookie 11150427020529159764 [11464.732447] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11464.733164] LustreError: 347165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11465.386287] Lustre: server umount lustre-MDT0001 complete [11480.398430] Lustre: server umount lustre-OST0000 complete [11496.166468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.802542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11500.621305] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11508.438306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.181596] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11522.405675] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11529.910665] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11536.588855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11542.277498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11542.891727] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11550.292796] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11559.924590] Lustre: server umount lustre-OST0000 complete [11565.026810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11565.033703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.039964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11565.045368] Lustre: Skipped 2 previous similar messages [11565.059105] Lustre: Skipped 1 previous similar message [11573.301222] Lustre: server umount lustre-MDT0000 complete [11575.268911] LustreError: 353549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.288289] LustreError: 353549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11576.289440] Lustre: 340679:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789590096/real 1789590096] req@00000000e125e663 x1876520728671488/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789590112 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11576.308445] Lustre: 340679:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11576.535469] LustreError: 353530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590112 with bad export cookie 11150427020529160513 [11576.539475] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11576.552685] LustreError: 353530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11576.801771] Lustre: server umount lustre-MDT0001 complete [11588.526604] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [11590.664900] Key type lgssc unregistered [11590.866687] LNet: 357080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11590.872109] LNetError: 357080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11590.886564] LNet: Removed LNI 192.168.204.152@tcp [11591.555590] Key type .llcrypt unregistered [11591.559164] Key type ._llcrypt unregistered [11613.685732] Key type ._llcrypt registered [11613.688526] Key type .llcrypt registered [11613.796599] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [11614.591262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11614.665522] alg: No test for adler32 (adler32-zlib) [11615.657759] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11615.893830] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [11617.552306] Key type lgssc registered [11618.453755] Lustre: Echo OBD driver; http://www.lustre.org/ [11625.537250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11625.563946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.939438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11630.705687] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11637.220430] LustreError: 358573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11637.234427] LustreError: 358573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11638.431928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11638.757852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11642.363943] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11648.828711] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11653.948518] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11660.490674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11660.779729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11666.366639] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11672.058162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11672.647332] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11682.284142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11682.287707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.301364] Lustre: Skipped 1 previous similar message [11687.394446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.408588] Lustre: Skipped 2 previous similar messages [11688.340042] Lustre: server umount lustre-MDT0000 complete [11691.685053] LustreError: 358553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590228 with bad export cookie 4934404881143539669 [11691.692354] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11691.700797] LustreError: 358553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11691.998265] Lustre: server umount lustre-MDT0001 complete [11705.620874] Lustre: server umount lustre-OST0000 complete [11715.725563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.171326] LustreError: 362283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11716.250189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11720.155895] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11721.698901] LustreError: 362284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.798549] LustreError: 362283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.191859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.776451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.695904] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11741.330067] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11747.659847] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11754.770520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11755.059483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11761.207549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11761.262453] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11767.504945] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11781.089408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11781.095678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11781.109426] Lustre: Skipped 1 previous similar message [11781.117727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11781.131100] Lustre: Skipped 2 previous similar messages [11784.857329] Lustre: server umount lustre-MDT0000 complete [11786.722725] LustreError: 362278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.738309] LustreError: 362278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11788.383159] LustreError: 362264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590324 with bad export cookie 4934404881143540782 [11788.393042] LustreError: 362264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11788.403607] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11788.726499] Lustre: server umount lustre-MDT0001 complete [11803.135548] Lustre: server umount lustre-OST0000 complete [11817.852691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11818.217036] LustreError: 366195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.291601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11822.060919] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11830.490275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.191299] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11842.008863] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11846.836686] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11852.998670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11853.378290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11853.384159] Lustre: Skipped 1 previous similar message [11858.972486] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [11863.603375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11865.778675] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11871.714158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11871.724408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11871.741302] Lustre: Skipped 2 previous similar messages [11871.750324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11871.760048] Lustre: Skipped 2 previous similar messages [11872.277652] Lustre: server umount lustre-OST0000 complete [11876.833644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11876.838816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11876.849941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11886.141799] Lustre: server umount lustre-MDT0000 complete [11889.125497] LustreError: 366923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.142437] LustreError: 366923:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11889.882614] LustreError: 366176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590426 with bad export cookie 4934404881143541797 [11889.884127] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11889.888951] LustreError: 366176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11890.157139] Lustre: server umount lustre-MDT0001 complete [11904.848603] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [11909.369418] Key type lgssc unregistered [11909.774713] LNet: 369724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11909.781924] LNetError: 369724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11910.829356] LNet: Removed LNI 192.168.204.152@tcp [11912.068086] Key type .llcrypt unregistered [11912.071047] Key type ._llcrypt unregistered [11935.429095] Key type ._llcrypt registered [11935.431455] Key type .llcrypt registered [11935.557442] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [11936.614651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11936.625277] alg: No test for adler32 (adler32-zlib) [11937.884515] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11938.157679] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [11939.825044] Key type lgssc registered [11941.547088] Lustre: Echo OBD driver; http://www.lustre.org/ [11952.507856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11952.528774] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.540237] LustreError: 371160:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11952.564980] LustreError: 371160:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11952.572286] LustreError: 371160:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11952.583611] LustreError: 371160:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11952.592450] LustreError: 371160:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11961.366783] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11961.371471] LustreError: 371558:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11961.380223] LustreError: 371558:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11961.384729] LustreError: 371558:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11961.389989] LustreError: 371558:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11961.405096] LustreError: 371558:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11982.615767] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [11985.532907] Key type lgssc unregistered [11985.895734] LNet: 372561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11985.902137] LNetError: 372561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11985.934606] LNet: Removed LNI 192.168.204.152@tcp [11986.850621] Key type .llcrypt unregistered [11986.855804] Key type ._llcrypt unregistered [12007.478481] Key type ._llcrypt registered [12007.480562] Key type .llcrypt registered [12007.599244] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [12026.275257] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [12027.458179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12027.470517] alg: No test for adler32 (adler32-zlib) [12028.566317] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12028.799781] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [12030.448285] Key type lgssc registered [12031.785353] Lustre: Echo OBD driver; http://www.lustre.org/ [12080.070650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12080.095355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12081.374455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12081.406526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12081.520493] Lustre: lustre-MDT0000: new disk, initializing [12081.597623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12081.615736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12085.556157] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12096.780846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.910499] Lustre: 375716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12096.982592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12096.986156] Lustre: Skipped 1 previous similar message [12097.079513] Lustre: lustre-MDT0001: new disk, initializing [12097.182273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12097.229621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12097.242194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12101.862450] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12111.639088] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12117.445994] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12125.916959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12126.129498] Lustre: lustre-OST0000: new disk, initializing [12126.133490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12126.144109] Lustre: 377685:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12126.274842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12133.068732] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12133.527567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12133.538790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12133.650608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12144.156934] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12148.745382] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12148.943416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12153.922779] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12154.156562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12159.459656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12159.460118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12159.465760] LustreError: Skipped 1 previous similar message [12159.467364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12159.497108] Lustre: Skipped 1 previous similar message [12162.643285] Lustre: server umount lustre-OST0000 complete [12169.190447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12169.202722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12169.227765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.243645] Lustre: Skipped 1 previous similar message [12173.256566] Lustre: server umount lustre-MDT0000 complete [12177.516562] LustreError: 376521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590713 with bad export cookie 7174583398220662848 [12177.539734] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12177.946388] Lustre: server umount lustre-MDT0001 complete [12188.281371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12189.931828] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:32:04 (1789590724) [12197.037260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.654477] LustreError: 380145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.690532] LustreError: 380145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12197.783339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12202.551625] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12202.977962] LustreError: 380146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.107570] LustreError: 380145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.007812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12212.476667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12218.220795] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12227.579495] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12234.125560] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12241.545358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12241.971851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12249.761874] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12259.134499] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12267.162237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12267.382546] Lustre: lustre-OST0001: new disk, initializing [12267.385474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12267.398289] Lustre: 382987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12267.537136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12274.097797] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12275.332803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12275.345546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12275.460909] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12284.752400] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12291.049962] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12291.051616] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.065770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12291.077577] Lustre: Skipped 2 previous similar messages [12291.100347] Lustre: Skipped 2 previous similar messages [12294.959554] Lustre: server umount lustre-OST0001 complete [12296.178251] LustreError: 381968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.197962] LustreError: 381968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12301.288702] LustreError: 381966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.318802] LustreError: 381966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12311.529218] LustreError: 381966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.560652] LustreError: 381966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12313.569663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12313.581483] LustreError: Skipped 1 previous similar message [12313.589265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.614176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12313.623958] Lustre: Skipped 1 previous similar message [12319.569567] Lustre: server umount lustre-OST0000 complete [12325.345760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12325.359270] LustreError: Skipped 1 previous similar message [12325.363916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.373123] Lustre: Skipped 1 previous similar message [12325.377314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12325.380695] Lustre: Skipped 3 previous similar messages [12330.717043] Lustre: server umount lustre-MDT0000 complete [12332.008237] LustreError: 380140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12332.034150] LustreError: 380140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12335.485547] LustreError: 380125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590871 with bad export cookie 7174583398220663730 [12335.493247] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12335.499528] LustreError: 380125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12342.016749] Lustre: server umount lustre-MDT0001 complete [12354.443489] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12357.810369] Key type lgssc unregistered [12358.108638] LNet: 384976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12358.121852] LNetError: 384976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12358.142150] LNet: Removed LNI 192.168.204.152@tcp [12359.195166] Key type .llcrypt unregistered [12359.198765] Key type ._llcrypt unregistered [12422.988275] Key type ._llcrypt registered [12422.993377] Key type .llcrypt registered [12423.124771] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [12424.220440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12424.355621] alg: No test for adler32 (adler32-zlib) [12425.338912] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12425.612917] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [12427.296946] Key type lgssc registered [12429.348301] Lustre: Echo OBD driver; http://www.lustre.org/ [12439.077179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12439.127545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12440.585170] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12440.644419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12440.857072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12445.388967] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12455.659960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12455.828723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12455.880309] Lustre: 387775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12456.158071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12462.048319] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12470.831441] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12476.824940] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12482.020838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12482.021623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12482.022666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12482.031136] Lustre: Skipped 1 previous similar message [12487.142560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12487.147642] Lustre: Skipped 1 previous similar message [12492.259655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12492.269655] Lustre: Skipped 1 previous similar message [12493.280152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12493.485240] Lustre: server umount lustre-MDT0000 complete [12497.555507] LustreError: 387766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591033 with bad export cookie 2964420736273204635 [12497.562723] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12497.571778] LustreError: 387766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12498.022147] Lustre: server umount lustre-MDT0001 complete [12509.044304] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:37:23 (1789591043) [12517.052918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12517.783467] LustreError: 390290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.796303] LustreError: 390290:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12517.857345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12522.978101] LustreError: 390291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12523.498885] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12528.099901] LustreError: 390290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12533.219182] LustreError: 390291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12533.226326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12533.639627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12538.965899] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12547.862901] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12554.290170] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12562.077151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12562.424766] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12562.649848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12569.687055] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12579.412211] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12589.091474] Lustre: Permanently deactivating lustre-OST0000 [12589.108246] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12599.344265] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12601.171793] LustreError: 393327:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12601.190832] LustreError: 393327:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12601.201937] LustreError: 393327:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12604.629306] LustreError: 393376:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12604.636446] LustreError: 393376:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12612.775278] Lustre: server umount lustre-OST0000 complete [12615.789160] LustreError: 393625:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12615.806685] LustreError: 393625:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12620.771239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12620.785922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12620.801810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12621.795023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12626.913295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12626.926884] Lustre: Skipped 3 previous similar messages [12628.016109] Lustre: server umount lustre-MDT0000 complete [12632.042107] LustreError: 391583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12632.070879] LustreError: 391583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12632.201823] LustreError: 390272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591168 with bad export cookie 2964420736273205216 [12632.215110] LustreError: 390272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12632.215317] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12641.990721] Lustre: server umount lustre-MDT0001 complete [12651.742556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.215465] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12660.052780] LustreError: 394785:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12666.508567] Lustre: 395125:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.152@tcp [12668.916282] Lustre: server umount MGS complete [12681.694452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.294993] LustreError: 395851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.334176] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12682.341192] Lustre: Skipped 1 previous similar message [12682.435427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12687.288938] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12697.663920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12697.999086] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12703.462846] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12711.817184] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12717.963254] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12719.950059] Lustre: Permanently reactivating lustre-OST0000 [12719.964341] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12719.974887] Lustre: Skipped 2 previous similar messages [12748.768273] LustreError: 397633:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12757.165205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12757.554385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12757.566414] Lustre: Skipped 1 previous similar message [12764.243399] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12774.077590] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12781.883995] Lustre: server umount lustre-OST0000 complete [12789.218601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.236992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12789.244785] Lustre: Skipped 1 previous similar message [12796.564547] Lustre: server umount lustre-MDT0000 complete [12799.461865] LustreError: 395851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.504261] LustreError: 395851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12800.420080] Lustre: 387095:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789591320/real 1789591320] req@0000000066ad2397 x1876522131624704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789591336 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12800.470300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.497657] Lustre: Skipped 1 previous similar message [12801.179846] LustreError: 397427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591337 with bad export cookie 2964420736273206266 [12801.185909] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12801.186853] LustreError: 397427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12801.555509] Lustre: server umount lustre-MDT0001 complete [12812.907193] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [12815.951593] Key type lgssc unregistered [12816.314896] LNet: 400214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12816.324905] LNetError: 400214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12816.349695] LNet: Removed LNI 192.168.204.152@tcp [12817.343697] Key type .llcrypt unregistered [12817.347061] Key type ._llcrypt unregistered [12837.132495] Key type ._llcrypt registered [12837.135595] Key type .llcrypt registered [12837.298619] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [12853.696399] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [12855.643248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12855.717484] alg: No test for adler32 (adler32-zlib) [12856.829873] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12857.199421] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [12858.896352] Key type lgssc registered [12860.171642] Lustre: Echo OBD driver; http://www.lustre.org/ [12899.364622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12899.405673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12900.823507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12900.903252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12901.034922] Lustre: lustre-MDT0000: new disk, initializing [12901.145910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12901.179866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12906.716562] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12917.945601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12918.097614] Lustre: 403368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12918.183185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12918.193644] Lustre: Skipped 1 previous similar message [12918.365825] Lustre: lustre-MDT0001: new disk, initializing [12918.513776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12918.580951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12918.605523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12924.408857] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12936.437659] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12944.259752] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12952.897921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12953.406905] Lustre: lustre-OST0000: new disk, initializing [12953.410157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12953.418302] Lustre: 405340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12953.517770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12960.820268] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [12963.451391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12963.465161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12963.553746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12974.653414] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12981.760654] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12982.029602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12987.484379] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12987.831094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12994.531640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12994.547589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12994.551206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12994.563622] Lustre: Skipped 1 previous similar message [12999.654789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12999.667392] Lustre: Skipped 2 previous similar messages [13004.256127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13004.461675] Lustre: server umount lustre-OST0000 complete [13009.892329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13009.909604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13009.915109] Lustre: Skipped 1 previous similar message [13009.945053] Lustre: Skipped 1 previous similar message [13015.010453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13015.016129] Lustre: Skipped 1 previous similar message [13015.294467] Lustre: server umount lustre-MDT0000 complete [13019.346988] LustreError: 403361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591555 with bad export cookie 16141847051303730756 [13019.363229] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13019.388739] LustreError: 403361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13019.954881] Lustre: server umount lustre-MDT0001 complete [13031.818463] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13034.637767] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:46:08 (1789591568) [13042.058514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13042.574319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13046.881326] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13047.778398] LustreError: 407802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.792388] LustreError: 407802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13051.878426] LustreError: 407801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.240406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13054.577166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13057.918720] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13066.570585] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13073.058197] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13080.349152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13080.724526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13086.989982] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13095.574777] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13097.649887] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13100.520407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13107.681668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13107.690197] LustreError: Skipped 1 previous similar message [13107.710669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13107.749774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13109.731065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13109.745920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13112.853824] Lustre: server umount lustre-OST0000 complete [13119.976241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13119.995840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13120.012219] Lustre: Skipped 1 previous similar message [13120.033877] Lustre: Skipped 3 previous similar messages [13123.839814] Lustre: server umount lustre-MDT0000 complete [13125.095526] LustreError: 408530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13125.129514] LustreError: 408530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13128.403239] LustreError: 407781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591664 with bad export cookie 16141847051303731638 [13128.403707] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13128.412278] LustreError: 407781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13129.057133] Lustre: server umount lustre-MDT0001 complete [13143.079796] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13146.533714] Key type lgssc unregistered [13146.895817] LNet: 411528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13146.906535] LNetError: 411528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13147.942469] LNet: Removed LNI 192.168.204.152@tcp [13149.173520] Key type .llcrypt unregistered [13149.177906] Key type ._llcrypt unregistered [13177.197706] Key type ._llcrypt registered [13177.201719] Key type .llcrypt registered [13177.312918] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13177.632439] Key type .llcrypt unregistered [13177.634831] Key type ._llcrypt unregistered [13199.433447] Key type ._llcrypt registered [13199.436976] Key type .llcrypt registered [13199.608076] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13201.485102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13201.558057] alg: No test for adler32 (adler32-zlib) [13202.839399] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [13203.136436] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [13204.920416] Key type lgssc registered [13206.191963] Lustre: Echo OBD driver; http://www.lustre.org/ [13214.870759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13214.885938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13216.360935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13221.142404] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13225.960660] LustreError: 413736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13225.973598] LustreError: 413736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13229.752846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13230.287669] LustreError: 413737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.325318] LustreError: 413737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13230.335979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13235.690360] LustreError: 413736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13235.723334] LustreError: 413736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13236.752413] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13240.808957] LustreError: 413737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13240.834165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13240.841411] LustreError: 413737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13244.443870] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13245.926664] LustreError: 413732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13245.957385] LustreError: 413732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13251.933716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13252.404211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13257.616630] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13266.408118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13266.418249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13266.431446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13267.939556] LustreError: 414462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13267.942971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13267.958155] LustreError: 414462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13267.977560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13268.105341] Lustre: server umount lustre-OST0000 complete [13273.059333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13273.062577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13273.069028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13273.069169] Lustre: Skipped 1 previous similar message [13274.429345] Lustre: server umount lustre-MDT0000 complete [13279.018711] LustreError: 413718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591815 with bad export cookie 11943468103988673477 [13279.020365] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13279.028931] LustreError: 413718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13279.429666] Lustre: server umount lustre-MDT0001 complete [13291.459310] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13294.900762] Key type lgssc unregistered [13295.268975] LNet: 416881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13295.285935] LNetError: 416881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13295.307838] LNet: Removed LNI 192.168.204.152@tcp [13296.182661] Key type .llcrypt unregistered [13296.188775] Key type ._llcrypt unregistered [13323.314575] Key type ._llcrypt registered [13323.319360] Key type .llcrypt registered [13323.435900] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13325.255786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13325.310352] alg: No test for adler32 (adler32-zlib) [13326.488814] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [13326.833985] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [13328.552236] Key type lgssc registered [13330.056157] Lustre: Echo OBD driver; http://www.lustre.org/ [13337.754572] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13337.766878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13353.376440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13358.496341] LustreError: 418267:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [13385.248194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13392.607540] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13402.390159] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13410.462400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.971439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13411.059368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13416.433696] LustreError: 418292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13416.474795] LustreError: 418292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13416.918360] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13420.528987] LustreError: 418293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13420.551925] LustreError: 418293:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13425.648466] LustreError: 418292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.680716] LustreError: 418292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13427.972500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13428.718493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13434.537505] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13444.550478] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13452.808222] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13462.505075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13462.515333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13462.541107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13464.561654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13464.567175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13464.567633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13466.799513] Lustre: server umount lustre-OST0000 complete [13474.785621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13474.812047] Lustre: Skipped 1 previous similar message [13474.812531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13474.844884] Lustre: Skipped 2 previous similar messages [13477.249046] Lustre: server umount lustre-MDT0000 complete [13479.916384] LustreError: 419310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13479.934793] LustreError: 419310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13481.305577] LustreError: 418273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789592017 with bad export cookie 4991406548623626750 [13481.329452] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13481.745744] Lustre: server umount lustre-MDT0001 complete [13493.857803] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13498.552511] Key type lgssc unregistered [13499.053855] LNet: 421840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13499.069755] LNetError: 421840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13499.105675] LNet: Removed LNI 192.168.204.152@tcp [13500.307849] Key type .llcrypt unregistered [13500.312056] Key type ._llcrypt unregistered [13536.537804] Key type ._llcrypt registered [13536.540550] Key type .llcrypt registered [13536.659664] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13538.782529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13538.898353] alg: No test for adler32 (adler32-zlib) [13540.169307] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [13540.642798] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [13542.344450] Key type lgssc registered [13543.561769] Lustre: Echo OBD driver; http://www.lustre.org/ [13552.720597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13552.733157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13554.157402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13559.006389] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13564.387491] LustreError: 423255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13564.406133] LustreError: 423255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13565.412181] LustreError: 423256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13568.345028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13568.627887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13573.197562] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13580.200284] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13585.476477] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13591.362537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13591.673593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13597.722873] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13599.870624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13605.224203] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13609.441658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13609.449060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13609.460325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13610.466645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13610.470780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13610.487930] Lustre: Skipped 1 previous similar message [13612.378550] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.52@tcp (stopping) [13612.390736] Lustre: Skipped 2 previous similar messages [13615.348199] Lustre: server umount lustre-MDT0000 complete [13615.586948] LustreError: 423255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13615.624920] LustreError: 423255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13618.025823] LustreError: 423235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.52@tcp arrived at 1789592154 with bad export cookie 2376312369875005257 [13619.913125] LustreError: 423237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789592156 with bad export cookie 2376312369875004361 [13619.925899] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13619.937119] LustreError: 423237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13620.706365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13620.707447] LustreError: 423256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13620.716822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13620.723893] LustreError: 423256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13624.739182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13624.739386] LustreError: 423255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13624.759956] LustreError: 423255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13626.398091] Lustre: server umount lustre-OST0000 complete [13634.700852] Lustre: server umount lustre-MDT0001 complete [13647.503945] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13649.991565] Key type lgssc unregistered [13650.381549] LNet: 426938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13650.388818] LNetError: 426938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13650.413478] LNet: Removed LNI 192.168.204.152@tcp [13651.304735] Key type .llcrypt unregistered [13651.307170] Key type ._llcrypt unregistered [13677.391313] Key type ._llcrypt registered [13677.395933] Key type .llcrypt registered [13677.512468] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13678.708902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13678.858041] alg: No test for adler32 (adler32-zlib) [13679.982367] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [13680.291943] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [13681.968219] Key type lgssc registered [13683.144045] Lustre: Echo OBD driver; http://www.lustre.org/ [13691.634377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13691.655351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13693.174568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13698.027615] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13703.653471] LustreError: 428352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13703.692051] LustreError: 428352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13707.791414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13708.243624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13712.852390] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13721.826474] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13729.622970] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13738.437189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13738.956094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13744.128346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13746.164428] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13755.543280] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13763.557782] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13763.576839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13766.919915] Lustre: server umount lustre-MDT0001 complete [13768.674358] LustreError: 428346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13768.687415] LustreError: 428346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13773.801681] LustreError: 428351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13773.802884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13773.845828] Lustre: Skipped 1 previous similar message [13773.854588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13773.866312] Lustre: Skipped 2 previous similar messages [13778.923494] LustreError: 428352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13778.919485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13779.414614] Lustre: server umount lustre-OST0000 complete [13783.356056] Lustre: server umount lustre-MDT0000 complete [13797.876107] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [13800.832806] Key type lgssc unregistered [13801.229885] LNet: 432038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13801.244749] LNetError: 432038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13801.262335] LNet: Removed LNI 192.168.204.152@tcp [13802.188735] Key type .llcrypt unregistered [13802.195706] Key type ._llcrypt unregistered [13829.946244] Key type ._llcrypt registered [13829.948700] Key type .llcrypt registered [13830.089860] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [13846.522280] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13847.365156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13847.548339] alg: No test for adler32 (adler32-zlib) [13848.585890] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [13848.729225] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [13850.452772] Key type lgssc registered [13851.452825] Lustre: Echo OBD driver; http://www.lustre.org/ [13896.353714] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [13912.389980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13912.418796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13913.794252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13913.887724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13914.051702] Lustre: lustre-MDT0000: new disk, initializing [13914.146674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13914.185197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13919.388918] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13938.055614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13938.408895] Lustre: lustre-OST0000: new disk, initializing [13938.411841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13938.414839] Lustre: Skipped 1 previous similar message [13938.427228] Lustre: 437080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13938.515128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13946.004390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13946.023628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13946.068934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13946.547773] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13961.336286] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13980.292084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13980.429084] Lustre: 436053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13980.528795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13980.678883] Lustre: lustre-MDT0001: new disk, initializing [13980.858413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13980.907965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13980.918282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13985.878178] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [13997.053240] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13997.288783] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14002.171647] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14002.347802] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14009.698171] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.52@tcp (stopping) [14011.880516] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14011.887718] Lustre: Skipped 1 previous similar message [14011.888417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14011.888489] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14011.897587] Lustre: Skipped 1 previous similar message [14013.597849] Lustre: server umount lustre-MDT0001 complete [14019.885751] LustreError: 436057:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14019.944432] LustreError: 436057:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14022.115203] LustreError: 436058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14022.134919] LustreError: 436058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14027.235052] LustreError: 437088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14027.266077] LustreError: 437088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14032.356882] LustreError: 437088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14032.375414] LustreError: 437088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14037.480201] LustreError: 436063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14037.480807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14037.501580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14039.837234] Lustre: server umount lustre-OST0000 complete [14050.221650] Lustre: server umount lustre-MDT0000 complete [14070.112192] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [14073.024953] Key type lgssc unregistered [14073.383077] LNet: 440831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14073.400957] LNetError: 440831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14074.473418] LNet: Removed LNI 192.168.204.152@tcp [14075.151938] Key type .llcrypt unregistered [14075.153448] Key type ._llcrypt unregistered [14120.245425] Key type ._llcrypt registered [14120.248324] Key type .llcrypt registered [14120.372065] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [14120.800474] Key type .llcrypt unregistered [14120.805866] Key type ._llcrypt unregistered [14139.136480] Key type ._llcrypt registered [14139.142743] Key type .llcrypt registered [14139.348751] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [14140.861208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14140.932626] alg: No test for adler32 (adler32-zlib) [14142.646741] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [14142.982315] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [14144.688535] Key type lgssc registered [14146.204747] Lustre: Echo OBD driver; http://www.lustre.org/ [14174.940974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14174.998440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14176.234740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14176.273747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14176.374560] Lustre: lustre-MDT0000: new disk, initializing [14176.457628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14176.473847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14181.723888] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [14192.316256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14192.458056] Lustre: 444199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14192.512369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14192.516310] Lustre: Skipped 1 previous similar message [14192.593962] Lustre: lustre-MDT0001: new disk, initializing [14192.668658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14192.699985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14192.712862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14197.933928] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [14209.372703] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14216.665675] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14225.676036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14226.158617] Lustre: lustre-OST0000: new disk, initializing [14226.165904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14226.183725] Lustre: 446170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14226.288194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14234.460603] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [14235.821310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14235.852914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14236.016896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14246.505575] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14254.049720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14254.056824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14254.069396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14256.613542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14256.616868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14256.623306] Lustre: Skipped 1 previous similar message [14256.640330] Lustre: Skipped 2 previous similar messages [14258.767882] Lustre: server umount lustre-MDT0000 complete [14262.849781] LustreError: 444193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789592799 with bad export cookie 15836975998231880962 [14262.851103] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14262.886898] LustreError: 444193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14263.237958] Lustre: server umount lustre-MDT0001 complete [14277.146700] Lustre: server umount lustre-OST0000 complete [14291.497472] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:07:05 (1789592825) [14302.351029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14302.875976] LustreError: 448200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14302.901035] LustreError: 448200:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14302.991059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14308.294561] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [14308.328539] LustreError: 448201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14313.446962] LustreError: 448200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14317.496283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14317.760847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14321.828380] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [14331.095257] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14337.749434] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14358.959597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14359.168158] Lustre: Found index 0 for lustre-OST0000, updating log [14359.396691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14366.136651] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [14373.760434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14376.562772] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14385.123539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14385.133637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14385.148670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14390.936761] Lustre: server umount lustre-OST0000 complete [14410.209333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14410.224103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14410.245812] Lustre: Skipped 1 previous similar message [14410.249446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14410.257736] Lustre: Skipped 3 previous similar messages [14414.540556] Lustre: server umount lustre-MDT0000 complete [14418.410870] LustreError: 448928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14418.444452] LustreError: 448928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14418.477930] LNet: 1 local NIs in recovery (showing 1): 192.168.204.152@tcp [14419.658552] LustreError: 448180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789592956 with bad export cookie 15836975998231882355 [14419.684193] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14419.698715] LustreError: 448180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14420.189428] Lustre: server umount lustre-MDT0001 complete [14429.525970] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:09:24 (1789592964) [15053.359533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15053.697367] LustreError: 454616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15053.805448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15058.171752] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15058.936657] LustreError: 454617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15064.040090] LustreError: 454616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15066.085397] LustreError: 454617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15066.118049] LustreError: 454617:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15068.155164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15068.697648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15073.814643] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15082.261733] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15088.618768] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15109.354479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15109.594627] Lustre: Found index 0 for lustre-OST0000, updating log [15109.859938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15110.689356] LNet: 1 local NIs in recovery (showing 1): 192.168.204.152@tcp [15117.655984] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15126.389152] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15129.147555] Lustre: server umount lustre-OST0000 complete [15134.885527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15134.892253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15134.908336] Lustre: Skipped 2 previous similar messages [15134.929845] Lustre: Skipped 5 previous similar messages [15135.974502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15135.986974] Lustre: Skipped 1 previous similar message [15136.997735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15137.006049] Lustre: Skipped 1 previous similar message [15139.043237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15139.052144] Lustre: Skipped 3 previous similar messages [15140.291850] Lustre: server umount lustre-MDT0000 complete [15141.090532] LustreError: 457079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15141.110667] LustreError: 457079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15145.055741] LustreError: 454642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789593681 with bad export cookie 15836975998231883440 [15145.070516] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15145.082723] LustreError: 454642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15145.618437] Lustre: server umount lustre-MDT0001 complete [15171.168310] LNet: 1 local NIs in recovery (showing 1): 192.168.204.152@tcp [15180.611518] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [15191.809605] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [15239.007714] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:22:53 (1789593773) [15259.090297] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [15262.146321] Key type lgssc unregistered [15262.476074] LNet: 461232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15262.496490] LNetError: 461232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15370.022285] LNet: Removed LNI 192.168.204.152@tcp [15371.111852] Key type .llcrypt unregistered [15371.113570] Key type ._llcrypt unregistered [15390.626804] Key type ._llcrypt registered [15390.630108] Key type .llcrypt registered [15390.769752] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [15392.027665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15392.178127] alg: No test for adler32 (adler32-zlib) [15393.282617] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [15393.547640] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [15395.249538] Key type lgssc registered [15396.590323] Lustre: Echo OBD driver; http://www.lustre.org/ [15401.502782] LNet: Added LNI 192.168.204.152@tcp100 [8/256/0/180] [15402.356829] LNet: Added LNI 192.168.204.152@tcp101 [8/256/0/180] [15403.401455] LNet: Added LNI 192.168.204.152@tcp102 [8/256/0/180] [15404.309532] LNet: Added LNI 192.168.204.152@tcp103 [8/256/0/180] [15405.287715] LNet: Added LNI 192.168.204.152@tcp104 [8/256/0/180] [15406.132115] LNet: Added LNI 192.168.204.152@tcp105 [8/256/0/180] [15407.154337] LNet: Added LNI 192.168.204.152@tcp106 [8/256/0/180] [15408.060088] LNet: Added LNI 192.168.204.152@tcp107 [8/256/0/180] [15409.007094] LNet: Added LNI 192.168.204.152@tcp108 [8/256/0/180] [15409.769376] LNet: Added LNI 192.168.204.152@tcp109 [8/256/0/180] [15410.604644] LNet: Added LNI 192.168.204.152@tcp110 [8/256/0/180] [15411.651061] LNet: Added LNI 192.168.204.152@tcp111 [8/256/0/180] [15413.053317] LNet: Added LNI 192.168.204.152@tcp112 [8/256/0/180] [15414.583254] LNet: Added LNI 192.168.204.152@tcp113 [8/256/0/180] [15415.357996] LNet: Added LNI 192.168.204.152@tcp114 [8/256/0/180] [15416.377213] LNet: Added LNI 192.168.204.152@tcp115 [8/256/0/180] [15417.670833] LNet: Added LNI 192.168.204.152@tcp116 [8/256/0/180] [15418.561549] LNet: Added LNI 192.168.204.152@tcp117 [8/256/0/180] [15419.814530] LNet: Added LNI 192.168.204.152@tcp118 [8/256/0/180] [15420.949829] LNet: Added LNI 192.168.204.152@tcp119 [8/256/0/180] [15422.159951] LNet: Added LNI 192.168.204.152@tcp120 [8/256/0/180] [15423.716826] LNet: Added LNI 192.168.204.152@tcp121 [8/256/0/180] [15425.107605] LNet: Added LNI 192.168.204.152@tcp122 [8/256/0/180] [15426.377644] LNet: Added LNI 192.168.204.152@tcp123 [8/256/0/180] [15427.698653] LNet: Added LNI 192.168.204.152@tcp124 [8/256/0/180] [15428.795574] LNet: Added LNI 192.168.204.152@tcp125 [8/256/0/180] [15429.954329] LNet: Added LNI 192.168.204.152@tcp126 [8/256/0/180] [15431.177597] LNet: Added LNI 192.168.204.152@tcp127 [8/256/0/180] [15432.133915] LNet: Added LNI 192.168.204.152@tcp128 [8/256/0/180] [15433.363820] LNet: Added LNI 192.168.204.152@tcp129 [8/256/0/180] [15434.614440] LNet: Added LNI 192.168.204.152@tcp130 [8/256/0/180] [15435.724746] LNet: Added LNI 192.168.204.152@tcp131 [8/256/0/180] [15436.816377] LNet: Added LNI 192.168.204.152@tcp132 [8/256/0/180] [15437.700616] LNet: Added LNI 192.168.204.152@tcp133 [8/256/0/180] [15438.665728] LNet: Added LNI 192.168.204.152@tcp134 [8/256/0/180] [15439.613381] LNet: Added LNI 192.168.204.152@tcp135 [8/256/0/180] [15446.864297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15446.900972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15448.019248] Lustre: 464257:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.152@tcp: can't add NID 192.168.204.152@tcp130: rc = -75 [15448.054068] LNetError: 464279:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15448.072930] LNetError: Protocol error connecting to 192.168.204.152@tcp120 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15448.080513] LNetError: Skipped 4 previous similar messages [15448.175738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15448.199304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15448.265988] Lustre: lustre-MDT0000: new disk, initializing [15448.341074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15448.358280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15448.864380] LNet: 5 local NIs in recovery (showing 5): 192.168.204.152@tcp122, 192.168.204.152@tcp123, 192.168.204.152@tcp124, 192.168.204.152@tcp125, 192.168.204.152@tcp120 [15449.122100] LNetError: 464279:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15449.148948] LNetError: 464279:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15449.164746] LNetError: Protocol error connecting to 192.168.204.152@tcp101 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15451.937753] LNetError: 461714:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15451.958579] LNetError: 461714:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15451.980224] LNetError: Protocol error connecting to 192.168.204.152@tcp123 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15451.998721] LNetError: Skipped 6 previous similar messages [15453.252903] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15463.206481] LNetError: 464284:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15463.221443] LNetError: 464284:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15463.237368] LNetError: Protocol error connecting to 192.168.204.152@tcp123 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15465.752101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15465.958657] Lustre: 464276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15466.125814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15466.131639] Lustre: Skipped 1 previous similar message [15466.238367] Lustre: lustre-MDT0001: new disk, initializing [15466.275318] Lustre: 465077:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.152@tcp: can't add NID 192.168.204.152@tcp130: rc = -75 [15466.283343] Lustre: 465077:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15466.474052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15466.547170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15466.561138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15472.495649] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15478.569673] LNetError: 461715:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15478.587653] LNetError: Protocol error connecting to 192.168.204.152@tcp117 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15482.862835] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15486.754059] LNetError: 464285:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15486.775957] LNetError: 464285:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15486.788217] LNetError: Protocol error connecting to 192.168.204.152@tcp124 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15486.797807] LNetError: Skipped 2 previous similar messages [15489.631861] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15497.507842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15497.912758] Lustre: lustre-OST0000: new disk, initializing [15497.927269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15497.942838] Lustre: 466249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15498.073215] Lustre: 466254:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.152@tcp: can't add NID 192.168.204.152@tcp130: rc = -75 [15498.084481] Lustre: 466254:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15498.152224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15503.227252] LNetError: 464284:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15503.246331] LNetError: 464284:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15503.268682] LNetError: Protocol error connecting to 192.168.204.152@tcp117 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15503.284881] LNetError: Skipped 5 previous similar messages [15503.411517] Lustre: 466480:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.152@tcp: can't add NID 192.168.204.152@tcp130: rc = -75 [15503.422273] Lustre: 466480:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15503.505695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15503.528101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15503.706768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15506.157868] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15509.280667] LNet: 1 local NIs in recovery (showing 1): 192.168.204.152@tcp123 [15517.984956] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15529.446128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15529.452640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15529.460424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15532.552490] Lustre: server umount lustre-OST0000 complete [15538.019695] LNetError: 461712:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.152 [15538.039051] LNetError: 461712:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15538.050696] LNetError: Protocol error connecting to 192.168.204.152@tcp124 at host 192.168.204.152:988: is it running a compatible version of Lustre? [15538.068563] LNetError: Skipped 6 previous similar messages [15538.149467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15538.156537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15538.175643] Lustre: Skipped 1 previous similar message [15538.203557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15538.217157] Lustre: Skipped 1 previous similar message [15539.689652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15539.691658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15539.726074] Lustre: Skipped 1 previous similar message [15544.544339] Lustre: server umount lustre-MDT0000 complete [15549.079401] LustreError: 465086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594085 with bad export cookie 579097267371693986 [15549.087595] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15549.091249] LustreError: 465086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15549.597291] Lustre: server umount lustre-MDT0001 complete [15562.571426] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [15565.536568] Key type lgssc unregistered [15566.019140] LNet: 468164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15566.036857] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15566.042950] LNet: Removed LNI 192.168.204.152@tcp [15567.077318] LNet: Removed LNI 192.168.204.152@tcp100 [15567.084441] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15567.099970] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15567.108994] LNet: Removed LNI 192.168.204.152@tcp101 [15568.160197] LNet: Removed LNI 192.168.204.152@tcp102 [15568.168067] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15568.189375] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15568.205847] LNet: Removed LNI 192.168.204.152@tcp103 [15569.248281] LNet: Removed LNI 192.168.204.152@tcp104 [15570.272496] LNet: Removed LNI 192.168.204.152@tcp105 [15570.275157] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15570.283933] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15570.298567] LNet: Removed LNI 192.168.204.152@tcp106 [15571.360154] LNet: Removed LNI 192.168.204.152@tcp107 [15572.384766] LNet: Removed LNI 192.168.204.152@tcp108 [15573.408210] LNet: Removed LNI 192.168.204.152@tcp109 [15574.432492] LNet: Removed LNI 192.168.204.152@tcp110 [15574.463403] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15574.482794] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15574.498166] LNet: Removed LNI 192.168.204.152@tcp111 [15575.520281] LNet: Removed LNI 192.168.204.152@tcp112 [15576.544203] LNet: Removed LNI 192.168.204.152@tcp113 [15577.568184] LNet: Removed LNI 192.168.204.152@tcp114 [15578.592826] LNet: Removed LNI 192.168.204.152@tcp115 [15579.616471] LNet: Removed LNI 192.168.204.152@tcp116 [15580.640286] LNet: Removed LNI 192.168.204.152@tcp117 [15581.665782] LNet: Removed LNI 192.168.204.152@tcp118 [15582.688181] LNet: Removed LNI 192.168.204.152@tcp119 [15582.696770] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15582.703140] LNetError: 468164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15582.711177] LNet: Removed LNI 192.168.204.152@tcp120 [15583.777737] LNet: Removed LNI 192.168.204.152@tcp121 [15584.800142] LNet: Removed LNI 192.168.204.152@tcp122 [15584.804746] LNet: Removed LNI 192.168.204.152@tcp123 [15585.824320] LNet: Removed LNI 192.168.204.152@tcp124 [15586.848276] LNet: Removed LNI 192.168.204.152@tcp125 [15586.851985] LNet: Removed LNI 192.168.204.152@tcp126 [15586.853531] LNet: Removed LNI 192.168.204.152@tcp127 [15586.869487] LNet: Removed LNI 192.168.204.152@tcp128 [15586.882694] LNet: Removed LNI 192.168.204.152@tcp129 [15586.890976] LNet: Removed LNI 192.168.204.152@tcp130 [15586.899224] LNet: Removed LNI 192.168.204.152@tcp131 [15586.900903] LNet: Removed LNI 192.168.204.152@tcp132 [15586.909834] LNet: Removed LNI 192.168.204.152@tcp133 [15586.919973] LNet: Removed LNI 192.168.204.152@tcp134 [15586.962183] LNet: Removed LNI 192.168.204.152@tcp135 [15587.858638] Key type .llcrypt unregistered [15587.860659] Key type ._llcrypt unregistered [15619.066707] Key type ._llcrypt registered [15619.068602] Key type .llcrypt registered [15619.250394] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [15635.927853] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [15637.406316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15637.482091] alg: No test for adler32 (adler32-zlib) [15638.784921] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [15639.186458] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [15640.872747] Key type lgssc registered [15642.296723] Lustre: Echo OBD driver; http://www.lustre.org/ [15681.558537] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:30:16 (1789594216) [15704.147354] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [15707.933949] Key type lgssc unregistered [15708.417482] LNet: 472249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15708.435592] LNetError: 472249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15708.466299] LNet: Removed LNI 192.168.204.152@tcp [15709.793903] Key type .llcrypt unregistered [15709.798630] Key type ._llcrypt unregistered [15730.137611] Key type ._llcrypt registered [15730.142338] Key type .llcrypt registered [15730.319494] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [15731.823888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15731.874214] alg: No test for adler32 (adler32-zlib) [15732.949576] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [15733.253858] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [15734.992335] Key type lgssc registered [15736.503649] Lustre: Echo OBD driver; http://www.lustre.org/ [15747.847925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15747.876312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15749.317633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15749.374160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15749.492231] Lustre: lustre-MDT0000: new disk, initializing [15749.676424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15749.707269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15754.862944] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15765.949638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15766.115805] Lustre: 473461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15766.164633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15766.167991] Lustre: Skipped 1 previous similar message [15766.302496] Lustre: lustre-MDT0001: new disk, initializing [15766.419688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15766.491451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15766.506866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15772.139873] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15783.512817] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15790.851365] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15798.958266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15799.223748] Lustre: lustre-OST0000: new disk, initializing [15799.226534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15799.233293] Lustre: 475429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15799.388581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15807.285469] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [15808.191672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15808.209567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15808.335828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15818.678683] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15822.770995] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing lnet_if_list [15829.266730] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15829.297587] LNet: Added LNI 192.168.204.152@tcp10 [8/256/0/180] [15837.748872] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15837.841114] LNet: Added LNI 192.168.204.152@tcp11 [8/256/0/180] [15844.597935] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15844.632170] LNet: Added LNI 192.168.204.152@tcp12 [8/256/0/180] [15850.639497] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15850.707866] LNet: Added LNI 192.168.204.152@tcp13 [8/256/0/180] [15856.421727] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15856.461628] LNet: Added LNI 192.168.204.152@tcp14 [8/256/0/180] [15862.255387] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15862.295742] LNet: Added LNI 192.168.204.152@tcp15 [8/256/0/180] [15867.098736] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15867.123782] LNet: Added LNI 192.168.204.152@tcp16 [8/256/0/180] [15872.536905] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15872.557645] LNet: Added LNI 192.168.204.152@tcp17 [8/256/0/180] [15878.536295] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15878.584453] LNet: Added LNI 192.168.204.152@tcp18 [8/256/0/180] [15884.543957] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15884.589506] LNet: Added LNI 192.168.204.152@tcp19 [8/256/0/180] [15891.023607] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15891.084823] LNet: Added LNI 192.168.204.152@tcp20 [8/256/0/180] [15897.358757] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15897.408405] LNet: Added LNI 192.168.204.152@tcp21 [8/256/0/180] [15904.139120] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15904.185104] LNet: Added LNI 192.168.204.152@tcp22 [8/256/0/180] [15910.503390] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15910.550430] LNet: Added LNI 192.168.204.152@tcp23 [8/256/0/180] [15916.605589] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15916.628713] LNet: Added LNI 192.168.204.152@tcp24 [8/256/0/180] [15922.919197] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15922.943411] LNet: Added LNI 192.168.204.152@tcp25 [8/256/0/180] [15929.437457] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15929.475566] LNet: Added LNI 192.168.204.152@tcp26 [8/256/0/180] [15936.380745] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15936.416723] LNet: Added LNI 192.168.204.152@tcp27 [8/256/0/180] [15942.849544] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15942.919947] LNet: Added LNI 192.168.204.152@tcp28 [8/256/0/180] [15949.727375] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15949.766593] LNet: Added LNI 192.168.204.152@tcp29 [8/256/0/180] [15957.491157] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15957.557842] LNet: Added LNI 192.168.204.152@tcp30 [8/256/0/180] [15965.863849] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15965.935502] LNet: Added LNI 192.168.204.152@tcp31 [8/256/0/180] [15974.084378] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15974.107508] LNet: Added LNI 192.168.204.152@tcp32 [8/256/0/180] [15981.719740] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15981.753974] LNet: Added LNI 192.168.204.152@tcp33 [8/256/0/180] [15989.079966] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15989.114566] LNet: Added LNI 192.168.204.152@tcp34 [8/256/0/180] [15997.213430] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15997.245465] LNet: Added LNI 192.168.204.152@tcp35 [8/256/0/180] [16005.878105] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16005.923043] LNet: Added LNI 192.168.204.152@tcp36 [8/256/0/180] [16013.916149] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16013.954796] LNet: Added LNI 192.168.204.152@tcp37 [8/256/0/180] [16022.248548] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16022.317565] LNet: Added LNI 192.168.204.152@tcp38 [8/256/0/180] [16031.292509] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16031.325724] LNet: Added LNI 192.168.204.152@tcp39 [8/256/0/180] [16039.771859] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16039.820377] LNet: Added LNI 192.168.204.152@tcp40 [8/256/0/180] [16048.685881] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16048.717049] LNet: Added LNI 192.168.204.152@tcp41 [8/256/0/180] [16057.081109] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16057.110226] LNet: Added LNI 192.168.204.152@tcp42 [8/256/0/180] [16064.533396] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16064.549446] LNet: Added LNI 192.168.204.152@tcp43 [8/256/0/180] [16071.441334] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16071.463656] LNet: Added LNI 192.168.204.152@tcp44 [8/256/0/180] [16080.358735] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16080.411719] LNet: Added LNI 192.168.204.152@tcp45 [8/256/0/180] [16090.881651] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16090.923212] LNet: Added LNI 192.168.204.152@tcp46 [8/256/0/180] [16100.307877] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16100.370045] LNet: Added LNI 192.168.204.152@tcp47 [8/256/0/180] [16109.962747] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16109.991156] LNet: Added LNI 192.168.204.152@tcp48 [8/256/0/180] [16120.903208] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16120.945793] LNet: Added LNI 192.168.204.152@tcp49 [8/256/0/180] [16131.575606] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16131.606258] LNet: Added LNI 192.168.204.152@tcp50 [8/256/0/180] [16155.636545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16155.646416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16155.674612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16156.647046] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16156.649433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16156.672096] Lustre: Skipped 1 previous similar message [16156.687759] Lustre: Skipped 2 previous similar messages [16160.600462] Lustre: server umount lustre-MDT0000 complete [16164.621066] LustreError: 473452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594700 with bad export cookie 1056630521390515140 [16164.632452] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16164.638806] LustreError: 473452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16164.853106] Lustre: server umount lustre-MDT0001 complete [16183.200153] Lustre: 472734:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594703/real 1789594703] req@00000000630912ae x1876525600397056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789594719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16183.227072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16186.336885] Lustre: 472731:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594706/real 1789594706] req@0000000005a2df00 x1876525600397184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789594722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16186.437729] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [16188.384230] Lustre: 472731:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594708/real 1789594708] req@000000003edf826a x1876525600397312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789594724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16191.392193] Lustre: 472734:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594711/real 1789594711] req@000000003bab5216 x1876525600397568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789594727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16193.955335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16196.576160] Lustre: 472733:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789594716/real 1789594716] req@0000000086fe37f5 x1876525600397696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789594732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16210.150227] LustreError: 475423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16210.184189] LustreError: 475423:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16210.225924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16212.264348] LustreError: 475424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16212.271585] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16215.551507] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16217.572855] LustreError: 475423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16222.696287] LustreError: 475424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16225.450676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16225.913618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16227.953622] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16230.285706] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16238.389766] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16245.374452] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16266.727071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16266.727870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16266.745201] LustreError: Skipped 1 previous similar message [16266.749582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16266.769194] Lustre: Skipped 1 previous similar message [16269.284631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16269.298291] Lustre: Skipped 2 previous similar messages [16272.396087] Lustre: server umount lustre-OST0000 complete [16276.965291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16276.977825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16277.001663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16282.463670] Lustre: server umount lustre-MDT0000 complete [16284.642767] LustreError: 499686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16284.660153] LustreError: 499686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16286.391816] LustreError: 473450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789594822 with bad export cookie 1056630521390517730 [16286.397323] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16286.401388] LustreError: 473450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16286.773665] Lustre: server umount lustre-MDT0001 complete [16299.182766] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [16302.163336] Key type lgssc unregistered [16302.549353] LNet: 502486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16302.559795] LNetError: 502486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16302.566194] LNet: Removed LNI 192.168.204.152@tcp [16302.573664] LNet: Removed LNI 192.168.204.152@tcp10 [16302.581414] LNet: Removed LNI 192.168.204.152@tcp11 [16302.587101] LNet: Removed LNI 192.168.204.152@tcp12 [16302.596878] LNet: Removed LNI 192.168.204.152@tcp13 [16302.603206] LNet: Removed LNI 192.168.204.152@tcp14 [16302.607249] LNet: Removed LNI 192.168.204.152@tcp15 [16302.618631] LNet: Removed LNI 192.168.204.152@tcp16 [16302.624847] LNet: Removed LNI 192.168.204.152@tcp17 [16302.627802] LNet: Removed LNI 192.168.204.152@tcp18 [16302.632403] LNet: Removed LNI 192.168.204.152@tcp19 [16302.642066] LNet: Removed LNI 192.168.204.152@tcp20 [16302.645048] LNet: Removed LNI 192.168.204.152@tcp21 [16302.649218] LNet: Removed LNI 192.168.204.152@tcp22 [16302.654327] LNet: Removed LNI 192.168.204.152@tcp23 [16302.657821] LNet: Removed LNI 192.168.204.152@tcp24 [16302.660929] LNet: Removed LNI 192.168.204.152@tcp25 [16302.667107] LNet: Removed LNI 192.168.204.152@tcp26 [16302.674257] LNet: Removed LNI 192.168.204.152@tcp27 [16302.680972] LNet: Removed LNI 192.168.204.152@tcp28 [16302.683587] LNet: Removed LNI 192.168.204.152@tcp29 [16302.697423] LNet: Removed LNI 192.168.204.152@tcp30 [16302.704258] LNet: Removed LNI 192.168.204.152@tcp31 [16302.710656] LNet: Removed LNI 192.168.204.152@tcp32 [16302.716071] LNet: Removed LNI 192.168.204.152@tcp33 [16302.719200] LNet: Removed LNI 192.168.204.152@tcp34 [16302.721970] LNet: Removed LNI 192.168.204.152@tcp35 [16302.732655] LNet: Removed LNI 192.168.204.152@tcp36 [16302.754165] LNet: Removed LNI 192.168.204.152@tcp37 [16302.756343] LNet: Removed LNI 192.168.204.152@tcp38 [16302.774101] LNet: Removed LNI 192.168.204.152@tcp39 [16302.787591] LNet: Removed LNI 192.168.204.152@tcp40 [16302.789612] LNet: Removed LNI 192.168.204.152@tcp41 [16302.798094] LNet: Removed LNI 192.168.204.152@tcp42 [16302.806201] LNet: Removed LNI 192.168.204.152@tcp43 [16302.826646] LNet: Removed LNI 192.168.204.152@tcp44 [16302.828515] LNet: Removed LNI 192.168.204.152@tcp45 [16302.843851] LNet: Removed LNI 192.168.204.152@tcp46 [16302.854733] LNet: Removed LNI 192.168.204.152@tcp47 [16302.856659] LNet: Removed LNI 192.168.204.152@tcp48 [16302.863991] LNet: Removed LNI 192.168.204.152@tcp49 [16302.893506] LNet: Removed LNI 192.168.204.152@tcp50 [16303.883489] Key type .llcrypt unregistered [16303.886223] Key type ._llcrypt unregistered [16324.713469] Key type ._llcrypt registered [16324.716785] Key type .llcrypt registered [16324.873227] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [16340.506572] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [16341.725192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16341.995905] alg: No test for adler32 (adler32-zlib) [16343.171893] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [16343.557336] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [16345.248450] Key type lgssc registered [16346.762190] Lustre: Echo OBD driver; http://www.lustre.org/ [16383.361146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16383.376411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16384.628514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16384.662564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16384.770312] Lustre: lustre-MDT0000: new disk, initializing [16384.891993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16384.914108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16390.472780] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16400.826095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16400.978968] Lustre: 505685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16401.033567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16401.036695] Lustre: Skipped 1 previous similar message [16401.096261] Lustre: lustre-MDT0001: new disk, initializing [16401.175224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16401.226423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16401.239069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16405.357351] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16414.028941] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16419.048913] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16425.546229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16425.723888] Lustre: *** cfs_fail_loc=90f, val=0*** [16425.786890] Lustre: lustre-OST0000: new disk, initializing [16425.790622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16425.796867] Lustre: 507701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16425.867655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16430.818960] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16433.748738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16433.765102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16433.864637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16439.943959] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16449.505715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16449.508179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16449.516804] LustreError: Skipped 1 previous similar message [16449.540495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16449.549124] Lustre: Skipped 1 previous similar message [16454.722570] Lustre: server umount lustre-OST0000 complete [16459.748909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16459.766338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16459.771559] Lustre: Skipped 2 previous similar messages [16459.781245] Lustre: Skipped 1 previous similar message [16464.873102] LustreError: 505693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16464.889272] LustreError: 505693:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16464.903930] Lustre: server umount lustre-MDT0000 complete [16468.859677] LustreError: 505679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595005 with bad export cookie 7210126060804234531 [16468.868167] LustreError: 505679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16468.869509] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16469.197379] Lustre: server umount lustre-MDT0001 complete [16481.136038] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [16485.083445] Key type lgssc unregistered [16485.528671] LNet: 509606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16485.551644] LNetError: 509606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16485.599201] LNet: Removed LNI 192.168.204.152@tcp [16487.010238] Key type .llcrypt unregistered [16487.013220] Key type ._llcrypt unregistered [16512.523779] Key type ._llcrypt registered [16512.526830] Key type .llcrypt registered [16512.644147] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [16513.733848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16513.742431] alg: No test for adler32 (adler32-zlib) [16514.874280] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [16515.128056] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [16516.915116] Key type lgssc registered [16517.982791] Lustre: Echo OBD driver; http://www.lustre.org/ [16527.981192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16528.026908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16529.910991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16536.570738] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16540.134449] LustreError: 510890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16540.146366] LustreError: 510890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16545.254044] LustreError: 510891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16545.360345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16545.936401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16551.546684] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16559.035620] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16565.285730] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16572.990133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16573.420189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16580.652086] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16589.188434] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16603.157635] loop0: detected capacity change from 0 to 204800000 [16603.316666] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16603.323306] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16603.335815] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16609.484952] loop0: detected capacity change from 0 to 204800000 [16609.671368] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16609.698578] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16609.721916] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16615.259973] loop0: detected capacity change from 0 to 204800000 [16621.300530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16621.658447] Lustre: fs15246-OST0000: new disk, initializing [16621.661799] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16621.668330] Lustre: 514487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16621.742264] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16627.520317] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16638.703926] loop1: detected capacity change from 0 to 204800000 [16644.288731] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16644.382130] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16644.437903] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16644.511158] Lustre: fs15246-MDT0000: new disk, initializing [16644.648451] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16644.668292] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16644.781224] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16645.796589] Lustre: 515611:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cfd804c6 x1876526420246144/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789595198 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16649.061547] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16654.901945] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16654.916197] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16654.945908] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16669.673371] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16669.689858] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16674.789337] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16675.576340] Lustre: server umount fs15246-MDT0000 complete [16679.915869] LustreError: 510885:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16684.628066] Lustre: server umount fs15246-OST0000 complete [16698.340767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16698.352331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16698.369184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16698.377617] Lustre: Skipped 1 previous similar message [16700.390594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16700.410581] Lustre: Skipped 1 previous similar message [16705.508254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16705.515251] Lustre: Skipped 1 previous similar message [16710.112988] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16710.434862] Lustre: server umount lustre-OST0000 complete [16714.727838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16714.734442] LustreError: Skipped 1 previous similar message [16714.742152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16714.754602] Lustre: Skipped 1 previous similar message [16714.757908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16720.633280] Lustre: server umount lustre-MDT0000 complete [16720.877415] LustreError: 511865:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16724.540225] LustreError: 510872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595260 with bad export cookie 1899094109102950748 [16724.544687] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16724.555132] LustreError: 510872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16724.995236] Lustre: server umount lustre-MDT0001 complete [16736.629715] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [16739.415482] Key type lgssc unregistered [16739.662572] LNet: 518710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16739.667829] LNetError: 518710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16739.684295] LNet: Removed LNI 192.168.204.152@tcp [16740.712860] Key type .llcrypt unregistered [16740.716676] Key type ._llcrypt unregistered [16760.687866] Key type ._llcrypt registered [16760.692437] Key type .llcrypt registered [16760.797949] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [16761.621901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16761.669504] alg: No test for adler32 (adler32-zlib) [16762.770278] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [16762.949530] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [16764.632830] Key type lgssc registered [16765.842754] Lustre: Echo OBD driver; http://www.lustre.org/ [16811.135749] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [16822.926066] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [16866.138245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16866.173535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16867.479142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16867.513169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16867.609796] Lustre: lustre-MDT0000: new disk, initializing [16867.771926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16867.807306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16872.528055] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16883.482276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16883.605377] Lustre: 523175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16883.683098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16883.687797] Lustre: Skipped 1 previous similar message [16883.749840] Lustre: lustre-MDT0001: new disk, initializing [16883.836889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16883.880787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16883.903922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16888.134991] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16897.263435] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16902.291661] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16909.287860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16909.576983] Lustre: lustre-OST0000: new disk, initializing [16909.584961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16909.599974] Lustre: 525144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16909.701277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16915.833528] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [16918.584484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16918.598195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16918.727862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16924.526601] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16929.183213] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16929.437888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16933.299906] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16933.552167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16939.489474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16939.496918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16939.525863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16939.531923] Lustre: Skipped 1 previous similar message [16941.601294] Lustre: server umount lustre-OST0000 complete [16949.729658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16949.741432] Lustre: Skipped 1 previous similar message [16949.748153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16949.754738] Lustre: Skipped 1 previous similar message [16953.293286] Lustre: server umount lustre-MDT0000 complete [16958.548249] LustreError: 523168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789595494 with bad export cookie 7265359215660125673 [16958.567126] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16958.570131] LustreError: 523168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16959.157522] Lustre: server umount lustre-MDT0001 complete [16988.562553] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [17000.664778] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [17038.379317] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16694 sec ========= 17:52:53 (1789595573) [17040.326422] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:52:54 (1789595574) === [17044.788631] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:52:59 (1789595579) === [17092.828454] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [17096.111895] Key type lgssc unregistered [17096.465878] LNet: 531793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17096.477749] LNetError: 531793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17096.492565] LNet: Removed LNI 192.168.204.152@tcp [17097.485254] Key type .llcrypt unregistered [17097.490820] Key type ._llcrypt unregistered