[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484704042 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002408] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010032] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.022151] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026604] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.032473] *** VALIDATE proc *** [ 0.034256] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038103] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.043819] debug: unmapping init [mem 0xffffffff8b059000-0xffffffff8b060fff] [ 0.045151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046645] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.055176] rcu: Hierarchical SRCU implementation. [ 0.057442] smp: Bringing up secondary CPUs ... [ 0.058617] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.064025] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.292250] node 0 deferred pages initialised in 220ms [ 0.296170] devtmpfs: initialized [ 0.298354] x86/mm: Memory block size: 128MB [ 0.302123] gcov: version magic: 0x41383552 [ 0.306338] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.307117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.309457] pinctrl core: initialized pinctrl subsystem [ 0.311327] [ 0.311952] ************************************************************* [ 0.315020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.317018] ** ** [ 0.320020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.322016] ** ** [ 0.325020] ** This means that this kernel is built to expose internal ** [ 0.327013] ** IOMMU data structures, which may compromise security on ** [ 0.329020] ** your system. ** [ 0.332018] ** ** [ 0.334015] ** If you see this message and you are not debugging the ** [ 0.337033] ** kernel, report this immediately to your vendor! ** [ 0.339016] ** ** [ 0.342021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.344015] ************************************************************* [ 0.347459] NET: Registered protocol family 16 [ 0.349492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.352080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.355084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.358560] cpuidle: using governor menu [ 0.360658] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.363620] PCI: Using configuration type 1 for base access [ 0.365149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.374150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.375021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.377224] cryptd: max_cpu_qlen set to 1000 [ 0.380327] ACPI: Added _OSI(Module Device) [ 0.382021] ACPI: Added _OSI(Processor Device) [ 0.384026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.386026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.391578] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.399440] ACPI: Interpreter enabled [ 0.401115] ACPI: PM: (supports S0 S3 S4 S5) [ 0.402019] ACPI: Using IOAPIC for interrupt routing [ 0.404170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.408410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.419054] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.420039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.422021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.426082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.431288] acpiphp: Slot [2] registered [ 0.433190] acpiphp: Slot [3] registered [ 0.435113] acpiphp: Slot [4] registered [ 0.436131] acpiphp: Slot [5] registered [ 0.438203] acpiphp: Slot [6] registered [ 0.440233] acpiphp: Slot [7] registered [ 0.442198] acpiphp: Slot [8] registered [ 0.443168] acpiphp: Slot [9] registered [ 0.446170] acpiphp: Slot [10] registered [ 0.447173] acpiphp: Slot [11] registered [ 0.449120] acpiphp: Slot [12] registered [ 0.451130] acpiphp: Slot [13] registered [ 0.452112] acpiphp: Slot [14] registered [ 0.454159] acpiphp: Slot [15] registered [ 0.456121] acpiphp: Slot [16] registered [ 0.457126] acpiphp: Slot [17] registered [ 0.459131] acpiphp: Slot [18] registered [ 0.460142] acpiphp: Slot [19] registered [ 0.462184] acpiphp: Slot [20] registered [ 0.463123] acpiphp: Slot [21] registered [ 0.464125] acpiphp: Slot [22] registered [ 0.466126] acpiphp: Slot [23] registered [ 0.467136] acpiphp: Slot [24] registered [ 0.469144] acpiphp: Slot [25] registered [ 0.471115] acpiphp: Slot [26] registered [ 0.472103] acpiphp: Slot [27] registered [ 0.473122] acpiphp: Slot [28] registered [ 0.475097] acpiphp: Slot [29] registered [ 0.476097] acpiphp: Slot [30] registered [ 0.477081] acpiphp: Slot [31] registered [ 0.478058] PCI host bridge to bus 0000:00 [ 0.480018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.482028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.484025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.486023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.489024] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.491021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.493404] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.497311] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.502512] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.511836] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.518013] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.521020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.523021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.525018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.527500] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.530806] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.533043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.535562] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.540015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.553020] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.558873] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.563739] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.571015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.578017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.594014] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.607662] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.613017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.620019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.636030] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.646739] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.656029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.664032] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.683019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.691843] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.702016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.708015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.724023] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.736855] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.743017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.749021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.768017] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.782561] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.790022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.798020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.812026] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.824143] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.827521] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.829441] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.832445] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.835278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.839128] iommu: Default domain type: Passthrough [ 0.841575] SCSI subsystem initialized [ 0.843154] ACPI: bus type USB registered [ 0.845159] usbcore: registered new interface driver usbfs [ 0.847125] usbcore: registered new interface driver hub [ 0.849175] usbcore: registered new device driver usb [ 0.851212] pps_core: LinuxPPS API ver. 1 registered [ 0.852015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.855086] PTP clock support registered [ 0.858084] EDAC MC: Ver: 3.0.0 [ 0.859554] PCI: Using ACPI for IRQ routing [ 0.860621] NetLabel: Initializing [ 0.862018] NetLabel: domain hash size = 128 [ 0.864019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.866111] NetLabel: unlabeled traffic allowed by default [ 0.869051] vgaarb: loaded [ 0.870336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.872018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.878053] clocksource: Switched to clocksource kvm-clock [ 0.986886] VFS: Disk quotas dquot_6.6.0 [ 0.988441] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.991220] *** VALIDATE ramfs *** [ 0.992570] *** VALIDATE hugetlbfs *** [ 0.994583] pnp: PnP ACPI init [ 0.997059] pnp: PnP ACPI: found 6 devices [ 1.021741] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.024927] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.026601] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.028842] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.030918] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.033070] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.036205] NET: Registered protocol family 2 [ 1.038927] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.043955] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.047567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.052580] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.056276] TCP: Hash tables configured (established 65536 bind 65536) [ 1.059563] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.063088] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.066275] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069539] NET: Registered protocol family 1 [ 1.073667] RPC: Registered named UNIX socket transport module. [ 1.076036] RPC: Registered udp transport module. [ 1.077433] RPC: Registered tcp transport module. [ 1.079098] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081667] NET: Registered protocol family 44 [ 1.083311] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.086050] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.088578] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.091283] PCI: CLS 0 bytes, default 64 [ 1.093222] Unpacking initramfs... [ 2.525583] debug: unmapping init [mem 0xffff8fe67cc54000-0xffff8fe67ffbffff] [ 2.529818] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.532142] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.534947] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.044398] Initialise system trusted keyrings [ 3.046290] Key type blacklist registered [ 3.049254] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.058681] zbud: loaded [ 3.062780] *** VALIDATE nfs *** [ 3.064194] *** VALIDATE nfs4 *** [ 3.067842] pstore: using deflate compression [ 3.071173] Platform Keyring initialized [ 3.188364] NET: Registered protocol family 38 [ 3.190190] Key type asymmetric registered [ 3.192395] Asymmetric key parser 'x509' registered [ 3.194247] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.197650] io scheduler mq-deadline registered [ 3.199268] io scheduler kyber registered [ 3.200777] io scheduler bfq registered [ 3.202858] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.206421] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.209510] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.212558] ACPI: Power Button [PWRF] [ 3.304420] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.395824] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.577347] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.669654] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.866990] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.898270] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.932095] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.940416] Non-volatile memory driver v1.3 [ 3.942139] Linux agpgart interface v0.103 [ 3.979689] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.983724] vda: detected capacity change from 0 to 68526080 [ 3.999632] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.003181] vdb: detected capacity change from 0 to 1073741824 [ 4.019150] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.022311] vdc: detected capacity change from 0 to 2621440000 [ 4.039335] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.042160] vdd: detected capacity change from 0 to 2621440000 [ 4.056810] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.060072] vde: detected capacity change from 0 to 4294967296 [ 4.075141] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.078089] vdf: detected capacity change from 0 to 4294967296 [ 4.088388] libphy: Fixed MDIO Bus: probed [ 4.095519] usbcore: registered new interface driver usbserial_generic [ 4.098218] usbserial: USB Serial support registered for generic [ 4.100817] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.105491] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.107195] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.109926] mousedev: PS/2 mouse device common for all mice [ 4.113273] rtc_cmos 00:05: RTC can wake from S4 [ 4.115703] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.120132] rtc_cmos 00:05: registered as rtc0 [ 4.123580] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.123685] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.126833] intel_pstate: CPU model not supported [ 4.133722] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.135633] hid: raw HID events driver (C) Jiri Kosina [ 4.139424] usbcore: registered new interface driver usbhid [ 4.141714] usbhid: USB HID core driver [ 4.143475] drop_monitor: Initializing network drop monitor service [ 4.146092] Initializing XFRM netlink socket [ 4.148395] NET: Registered protocol family 10 [ 4.152075] Segment Routing with IPv6 [ 4.153677] NET: Registered protocol family 17 [ 4.156105] mpls_gso: MPLS GSO support [ 4.161861] RAS: Correctable Errors collector initialized. [ 4.164622] AVX version of gcm_enc/dec engaged. [ 4.166566] AES CTR mode by8 optimization enabled [ 4.250931] sched_clock: Marking stable (4250899892, 0)->(5176446974, -925547082) [ 4.254453] registered taskstats version 1 [ 4.256489] Loading compiled-in X.509 certificates [ 4.258758] zswap: loaded using pool lzo/zbud [ 4.287455] Key type big_key registered [ 4.300267] Key type encrypted registered [ 4.301745] ima: No TPM chip found, activating TPM-bypass! [ 4.304470] ima: Allocated hash algorithm: sha1 [ 4.306314] ima: No architecture policies found [ 4.308068] evm: Initialising EVM extended attributes: [ 4.309889] evm: security.selinux [ 4.311448] evm: security.ima [ 4.312779] evm: security.capability [ 4.314327] evm: HMAC attrs: 0x1 [ 4.316992] rtc_cmos 00:05: setting system clock to 2025-11-17 03:41:45 UTC (1763350905) [ 4.324260] debug: unmapping init [mem 0xffffffff8c003000-0xffffffff8c1fffff] [ 4.326500] debug: unmapping init [mem 0xffffffff8ad82000-0xffffffff8b058fff] [ 4.334184] Write protecting the kernel read-only data: 28672k [ 4.337849] debug: unmapping init [mem 0xffffffff89403000-0xffffffff895fffff] [ 4.340988] debug: unmapping init [mem 0xffffffff89d14000-0xffffffff89dfffff] [ 4.376362] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.386213] systemd[1]: Detected virtualization kvm. [ 4.387992] systemd[1]: Detected architecture x86-64. [ 4.389945] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.416346] systemd[1]: No hostname configured. [ 4.418221] systemd[1]: Set hostname to . [ 4.421769] random: systemd: uninitialized urandom read (16 bytes read) [ 4.424894] systemd[1]: Initializing machine ID from random generator. [ 4.470508] random: ln: uninitialized urandom read (6 bytes read) [ 4.555529] random: systemd: uninitialized urandom read (16 bytes read) [ 4.558743] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.562994] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.567452] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.236200] device-mapper: uevent: version 1.0.3 [ 5.240117] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.958852] virtio_net virtio0 ens2: renamed from eth0 [ 5.987114] random: fast init done [ 6.022451] scsi host0: ata_piix [ 6.048677] scsi host1: ata_piix [ 6.050191] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.052384] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.685418] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.836114] random: crng init done [ 10.837529] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.948332] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.317473] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.016936] SELinux: Disabled at runtime. [ 15.121283] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.132469] systemd[1]: Detected virtualization kvm. [ 15.134640] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.454818] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.459928] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.466303] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.498085] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.501649] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.516471] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.530549] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ 16.590071] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.588557] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.276851] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.354860] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.616360] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.831169] EDAC sbridge: Ver: 1.1.2 [ 22.773148] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 23.631546] NFS: Registering the id_resolver key type [ 23.633847] Key type id_resolver registered [ 23.635591] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg338-server login: [ 36.159645] hrtimer: interrupt took 3242793 ns [ 71.102407] libcfs: loading out-of-tree module taints kernel. [ 71.128579] Key type ._llcrypt registered [ 71.130227] Key type .llcrypt registered [ 71.233741] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 86.864661] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 87.967946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.975622] alg: No test for adler32 (adler32-zlib) [ 89.233125] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 89.773501] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 91.536220] Key type lgssc registered [ 93.013301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.535093] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 107.746926] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.316659] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.094275] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 128.322195] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.222154] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 154.096758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.159587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.185188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.401957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.440734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.521436] Lustre: lustre-MDT0000: new disk, initializing [ 155.638400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.654330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.795873] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.694065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.772903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 170.830258] Lustre: 6513:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.886674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.892667] Lustre: Skipped 1 previous similar message [ 170.991793] Lustre: lustre-MDT0001: new disk, initializing [ 171.075487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.091584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.100296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.214840] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.812283] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.781439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.832773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.014757] Lustre: lustre-OST0000: new disk, initializing [ 187.020737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.073511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.286533] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.532398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.547844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.581973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.339663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.450674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 203.551850] Lustre: lustre-OST0001: new disk, initializing [ 203.561730] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.652593] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.057206] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.008487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.015755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.143171] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.218646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.177285] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.695431] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.408205] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing yml_node [ 249.566870] Lustre: DEBUG MARKER: Client: 2.16.61.41 [ 251.725875] Lustre: DEBUG MARKER: MDS: 2.16.61.41 [ 254.308067] Lustre: DEBUG MARKER: OSS: 2.16.61.41 [ 255.894623] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:45:55 EST 2025 [ 273.367727] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 274.774305] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.860564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.864063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.867371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.910697] Lustre: Skipped 1 previous similar message [ 293.944199] Lustre: Skipped 3 previous similar messages [ 295.627859] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 295.721701] Lustre: server umount lustre-MDT0000 complete [ 299.556109] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351200 with bad export cookie 7430985807751685799 [ 299.562952] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.101345] LustreError: 12218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.101345] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.101357] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 304.104202] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.104212] Lustre: Skipped 2 previous similar messages [ 304.104670] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.114709] LustreError: 12218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 305.765399] LustreError: 12419:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 305.770475] LustreError: 12419:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 305.920571] Lustre: server umount lustre-MDT0001 complete [ 309.446902] LustreError: 12622:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 309.455652] LustreError: 12622:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 309.503153] Lustre: server umount lustre-OST0000 complete [ 312.941150] LustreError: 12823:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 312.949911] LustreError: 12823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 313.142841] Lustre: server umount lustre-OST0001 complete [ 319.368956] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 327.440792] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 335.459395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.349940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.650445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 352.011398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.198692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.245496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.478968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 358.503610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 358.610287] Lustre: lustre-MDT0000: new disk, initializing [ 358.678238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.708405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.633784] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 372.563773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 372.649537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 372.801114] Lustre: 15010:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 372.884405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 372.894250] Lustre: Skipped 1 previous similar message [ 372.986944] Lustre: lustre-MDT0001: new disk, initializing [ 373.077260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 373.135822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 373.149838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 376.638698] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 383.764185] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 387.973184] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 395.634964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 395.783623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 396.124793] Lustre: lustre-OST0000: new disk, initializing [ 396.130320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 396.229604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.548971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 397.561598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 397.701436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 402.676772] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 409.859257] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 414.084933] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 414.374369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 418.191175] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 418.364766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 423.397061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.403714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.405409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 423.405415] Lustre: Skipped 1 previous similar message [ 423.447023] Lustre: Skipped 2 previous similar messages [ 426.237163] LustreError: 17952:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 426.244379] LustreError: 17952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 426.445758] Lustre: server umount lustre-OST0000 complete [ 433.634134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.641560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 433.646769] Lustre: Skipped 2 previous similar messages [ 438.775567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.384146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.488339] LustreError: 18153:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 444.492064] LustreError: 18153:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 444.623496] Lustre: server umount lustre-MDT0000 complete [ 447.969741] LustreError: 15021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.003593] LustreError: 15021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 448.223927] LustreError: 15001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351349 with bad export cookie 7430985807751687332 [ 448.225352] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.235807] LustreError: 15001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 448.668774] Lustre: server umount lustre-MDT0001 complete [ 451.762210] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:49:11 (1763351351) [ 458.036246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.422768] LustreError: 18881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.496984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.778365] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 463.848396] LustreError: 18881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.944034] LustreError: 18881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.548178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.530496] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 483.349189] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 486.963796] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.347205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 498.563747] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 503.923252] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 513.514945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 513.519083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.532172] LustreError: Skipped 2 previous similar messages [ 513.542502] Lustre: Skipped 1 previous similar message [ 513.543689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 513.558619] Lustre: Skipped 4 previous similar messages [ 515.593480] LustreError: 21124:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 515.609109] LustreError: 21124:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 515.734527] Lustre: server umount lustre-OST0000 complete [ 519.137323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 523.748613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.754590] Lustre: Skipped 2 previous similar messages [ 525.291568] Lustre: server umount lustre-MDT0000 complete [ 528.866211] LustreError: 18876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.885748] LustreError: 18876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 528.999377] LustreError: 21328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351430 with bad export cookie 7430985807751688214 [ 529.006373] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.355598] Lustre: server umount lustre-MDT0001 complete [ 537.588344] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 540.046978] Key type lgssc unregistered [ 540.299832] LNet: 22060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.312774] LNetError: 22060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.334430] LNet: Removed LNI 192.168.203.138@tcp [ 541.047394] Key type .llcrypt unregistered [ 541.049091] Key type ._llcrypt unregistered [ 562.569988] Key type ._llcrypt registered [ 562.571608] Key type .llcrypt registered [ 562.690864] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 563.709177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 563.732479] alg: No test for adler32 (adler32-zlib) [ 564.918822] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 565.140106] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 566.808163] Key type lgssc registered [ 568.183101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 574.851452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 574.878054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.274404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.783106] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 586.724959] LustreError: 23463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.748742] LustreError: 23463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 587.609782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 588.133529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 591.684236] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 596.656969] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 600.351958] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 607.398825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 607.683423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.898638] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 618.121155] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 620.992850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 633.831821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 633.843298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.869778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 634.347345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 634.352238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.377964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.387089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.398254] Lustre: Skipped 1 previous similar message [ 637.900483] LustreError: 26152:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 638.074996] Lustre: server umount lustre-OST0000 complete [ 644.583490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.590424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.610436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.630368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.777875] LustreError: 26353:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 647.782361] LustreError: 26353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 647.923368] Lustre: server umount lustre-MDT0000 complete [ 651.321374] LustreError: 23443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351552 with bad export cookie 3330135758337301468 [ 651.330398] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.338354] LustreError: 23443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 651.476936] LustreError: 26556:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 651.479774] LustreError: 26556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 651.689691] Lustre: server umount lustre-MDT0001 complete [ 662.237918] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 664.695659] Key type lgssc unregistered [ 664.970729] LNet: 27089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.979240] LNetError: 27089:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.999819] LNet: Removed LNI 192.168.203.138@tcp [ 665.615278] Key type .llcrypt unregistered [ 665.617256] Key type ._llcrypt unregistered [ 684.813057] Key type ._llcrypt registered [ 684.814636] Key type .llcrypt registered [ 684.918613] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 685.889269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.912080] alg: No test for adler32 (adler32-zlib) [ 686.972602] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 687.154509] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 688.824922] Key type lgssc registered [ 689.748430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.604326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.638495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.268669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.828680] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 708.580823] LustreError: 28461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.596768] LustreError: 28461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 709.196042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 709.448617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 712.889218] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 717.522174] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 721.010748] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 734.066081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 734.377148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.181922] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 743.602870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 743.768575] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 749.032950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.035534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 749.035972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 749.048830] Lustre: Skipped 1 previous similar message [ 754.145577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 754.148493] Lustre: Skipped 1 previous similar message [ 759.268059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 763.363434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 763.418942] LustreError: 31049:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 763.556976] Lustre: server umount lustre-OST0000 complete [ 769.505114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.507222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.510878] Lustre: Skipped 1 previous similar message [ 769.536709] Lustre: Skipped 2 previous similar messages [ 773.550113] LustreError: 31252:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 773.563211] LustreError: 31252:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 773.665200] Lustre: server umount lustre-MDT0000 complete [ 774.626413] LustreError: 28461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 774.647148] LustreError: 28461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 776.741328] LustreError: 28443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351677 with bad export cookie 5245630930701019875 [ 776.754680] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.910297] LustreError: 31455:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 776.913341] LustreError: 31455:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 777.175695] Lustre: server umount lustre-MDT0001 complete [ 786.257883] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 788.763633] Key type lgssc unregistered [ 789.002632] LNet: 31988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.033030] LNetError: 31988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.067465] LNet: Removed LNI 192.168.203.138@tcp [ 789.966364] Key type .llcrypt unregistered [ 789.967642] Key type ._llcrypt unregistered [ 809.339248] Key type ._llcrypt registered [ 809.341032] Key type .llcrypt registered [ 809.460396] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 810.086980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.137356] alg: No test for adler32 (adler32-zlib) [ 811.249402] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 811.519871] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 813.184214] Key type lgssc registered [ 814.202943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 820.156569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 820.176268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.598542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.555640] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 830.747569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 830.945917] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 831.086681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.013212] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 838.294420] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 841.235338] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 846.733367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 846.820826] LustreError: 34961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.841114] LustreError: 34961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 847.015501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.317154] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 852.478319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 855.813345] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 862.689366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 862.695122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.704732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 866.789136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.792395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.796157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.358132] LustreError: 35608:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 868.418739] Lustre: server umount lustre-OST0000 complete [ 871.909462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.909638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 871.925530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 871.930034] Lustre: Skipped 2 previous similar messages [ 877.025685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 877.833841] LustreError: 35811:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 877.846898] LustreError: 35811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 877.933432] Lustre: server umount lustre-MDT0000 complete [ 880.964356] LustreError: 33343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351782 with bad export cookie 6868491077120038257 [ 880.970170] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 881.079876] LustreError: 36013:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 881.087591] LustreError: 36013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 881.206640] Lustre: server umount lustre-MDT0001 complete [ 887.995396] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 889.901953] Key type lgssc unregistered [ 890.111375] LNet: 36544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.115528] LNetError: 36544:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.125732] LNet: Removed LNI 192.168.203.138@tcp [ 890.642099] Key type .llcrypt unregistered [ 890.643884] Key type ._llcrypt unregistered [ 904.411950] Key type ._llcrypt registered [ 904.413238] Key type .llcrypt registered [ 904.478084] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 905.020285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.056087] alg: No test for adler32 (adler32-zlib) [ 905.989485] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 906.133036] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 907.744620] Key type lgssc registered [ 908.437909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.265587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.288674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.659359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.579487] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 925.155325] LustreError: 37918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.178435] LustreError: 37918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 925.234971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 925.605281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.747545] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 932.755104] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 935.248938] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 940.547472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 940.872751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.666668] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 949.049551] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 960.390993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 965.606048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.611360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.614386] Lustre: Skipped 1 previous similar message [ 965.618103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.636118] LustreError: 40183:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 970.735879] Lustre: server umount lustre-OST0000 complete [ 987.105474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 987.113258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.129954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 987.138031] Lustre: Skipped 2 previous similar messages [ 990.178657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.178919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.050176] LustreError: 40560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 993.060025] LustreError: 40560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 993.139612] Lustre: server umount lustre-MDT0000 complete [ 995.297676] LustreError: 37919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.314286] LustreError: 37919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 995.666168] LustreError: 37899:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351896 with bad export cookie 12469579533753203816 [ 995.670459] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.672163] LustreError: 37899:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1001.827700] LustreError: 40762:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1001.833283] LustreError: 40762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1001.960479] Lustre: server umount lustre-MDT0001 complete [ 1009.151739] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1011.009188] Key type lgssc unregistered [ 1011.187163] LNet: 41295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.193697] LNetError: 41295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.202552] LNet: Removed LNI 192.168.203.138@tcp [ 1011.804442] Key type .llcrypt unregistered [ 1011.806432] Key type ._llcrypt unregistered [ 1025.740296] Key type ._llcrypt registered [ 1025.743805] Key type .llcrypt registered [ 1025.819913] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1026.358316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.429540] alg: No test for adler32 (adler32-zlib) [ 1027.364075] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1027.483370] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1029.112179] Key type lgssc registered [ 1029.804599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.800521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.813061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.145939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.637676] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1044.433345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1044.653233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.250595] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1050.838627] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1053.232098] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1058.102432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1058.314176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.438340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1062.303457] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1065.892554] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1075.684323] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.689280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.693904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.405753] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 1077.410200] Lustre: Skipped 2 previous similar messages [ 1079.970115] LustreError: 44912:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1080.054521] Lustre: server umount lustre-MDT0000 complete [ 1080.832496] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.840835] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1082.511980] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.136103] LustreError: 44286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351984 with bad export cookie 3452387297529152499 [ 1083.140349] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.148297] LustreError: 44286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1085.921124] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.921550] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.950795] Lustre: Skipped 2 previous similar messages [ 1085.957565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1088.928931] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1088.932640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1088.945641] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1088.955424] Lustre: Skipped 1 previous similar message [ 1089.385637] LustreError: 45116:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1089.388794] LustreError: 45116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1089.469680] Lustre: server umount lustre-MDT0001 complete [ 1117.153344] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1117.184469] LustreError: 45339:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1117.188032] LustreError: 45339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1117.276718] Lustre: server umount lustre-OST0000 complete [ 1127.039442] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1128.837705] Key type lgssc unregistered [ 1129.060174] LNet: 46178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.064226] LNetError: 46178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.076393] LNet: Removed LNI 192.168.203.138@tcp [ 1129.678755] Key type .llcrypt unregistered [ 1129.680450] Key type ._llcrypt unregistered [ 1144.832398] Key type ._llcrypt registered [ 1144.834341] Key type .llcrypt registered [ 1144.893730] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1145.503247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1145.525228] alg: No test for adler32 (adler32-zlib) [ 1146.494562] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1146.657058] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1148.296190] Key type lgssc registered [ 1149.125898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1154.532540] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1154.546587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1170.016424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1175.136226] LustreError: 47522:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 1201.825829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.034523] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1209.984799] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1376.619097] LustreError: 48243:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1376.702212] Lustre: server umount lustre-OST0000 complete [ 1388.914746] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1390.531638] Key type lgssc unregistered [ 1390.686375] LNet: 49080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.692188] LNetError: 49080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.704616] LNet: Removed LNI 192.168.203.138@tcp [ 1391.209272] Key type .llcrypt unregistered [ 1391.211394] Key type ._llcrypt unregistered [ 1404.894049] Key type ._llcrypt registered [ 1404.896715] Key type .llcrypt registered [ 1404.965092] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1405.491515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.541705] alg: No test for adler32 (adler32-zlib) [ 1406.463574] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1406.583491] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1408.200440] Key type lgssc registered [ 1408.806691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.359888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.370435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.624962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.268115] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1422.233612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1422.451946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.744911] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1428.001489] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.133649] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1434.217446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1434.435183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.184692] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1439.540964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1442.174918] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1444.837988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.840260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.864791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.766221] LustreError: 52694:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1445.837577] Lustre: server umount lustre-OST0000 complete [ 1452.003400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.007359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.016825] Lustre: Skipped 1 previous similar message [ 1454.195422] LustreError: 52896:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1454.198100] LustreError: 52896:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1454.239695] Lustre: server umount lustre-MDT0000 complete [ 1456.881416] LustreError: 50434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352358 with bad export cookie 4663135705435047204 [ 1456.887156] LustreError: 50434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1456.887381] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.979887] LustreError: 53097:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1456.982158] LustreError: 53097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1457.108584] Lustre: server umount lustre-MDT0001 complete [ 1464.002674] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1465.491937] Key type lgssc unregistered [ 1465.681140] LNet: 53629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.685672] LNetError: 53629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.694355] LNet: Removed LNI 192.168.203.138@tcp [ 1466.120345] Key type .llcrypt unregistered [ 1466.122089] Key type ._llcrypt unregistered [ 1479.878785] Key type ._llcrypt registered [ 1479.880372] Key type .llcrypt registered [ 1479.950035] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1480.510265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1480.565666] alg: No test for adler32 (adler32-zlib) [ 1481.542645] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1481.677843] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1483.312174] Key type lgssc registered [ 1484.044924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1489.030529] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1489.043130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1504.544289] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1509.665554] LustreError: 54975:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 1536.353313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.342654] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1544.230650] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1548.885296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.202133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.252516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1552.049145] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1556.451321] LustreError: 56081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.458847] LustreError: 56081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1557.982751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1558.216254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1560.784118] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1564.441637] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1566.979820] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1568.744549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.745239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1568.753491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.861486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.865290] Lustre: Skipped 1 previous similar message [ 1574.456974] LustreError: 57248:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1574.540816] Lustre: server umount lustre-OST0000 complete [ 1594.337743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.340317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1594.340728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.340732] Lustre: Skipped 1 previous similar message [ 1594.346563] Lustre: Skipped 2 previous similar messages [ 1598.752334] LustreError: 57626:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1598.756267] LustreError: 57626:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1598.866659] Lustre: server umount lustre-MDT0000 complete [ 1599.461106] LustreError: 56081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.501272] LustreError: 56081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1601.798612] LustreError: 54982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352502 with bad export cookie 12892165840374279033 [ 1601.816649] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.352187] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1616.432059] LustreError: 57829:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1616.438981] LustreError: 57829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1616.541147] Lustre: server umount lustre-MDT0001 complete [ 1623.214271] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1624.813545] Key type lgssc unregistered [ 1625.064703] LNet: 58363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.068956] LNetError: 58363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.081220] LNet: Removed LNI 192.168.203.138@tcp [ 1625.669979] Key type .llcrypt unregistered [ 1625.672735] Key type ._llcrypt unregistered [ 1640.378647] Key type ._llcrypt registered [ 1640.380076] Key type .llcrypt registered [ 1640.446683] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1641.085229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.115922] alg: No test for adler32 (adler32-zlib) [ 1642.134770] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1642.310045] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1643.944260] Key type lgssc registered [ 1644.746426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.417306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1650.440318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.773509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.742896] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1660.903708] LustreError: 59734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.919776] LustreError: 59734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1661.261770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1661.544802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1664.423147] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1668.383729] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1670.848401] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1675.785389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1676.013117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.047294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1679.793333] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1682.977753] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1701.863301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.870784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1701.870832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1701.879243] Lustre: Skipped 1 previous similar message [ 1704.052144] LustreError: 62002:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1704.114771] Lustre: server umount lustre-OST0000 complete [ 1706.979771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.990509] Lustre: Skipped 1 previous similar message [ 1706.991693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.999079] Lustre: Skipped 2 previous similar messages [ 1712.099623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.866138] LustreError: 62203:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1712.870790] LustreError: 62203:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1712.913176] Lustre: server umount lustre-MDT0000 complete [ 1715.268750] LustreError: 59713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352616 with bad export cookie 17001702234197945646 [ 1715.272182] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.277774] LustreError: 59713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1715.389677] LustreError: 62404:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1715.392514] LustreError: 62404:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1715.500460] Lustre: server umount lustre-MDT0001 complete [ 1721.568415] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1723.467923] Key type lgssc unregistered [ 1723.687824] LNet: 62936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.691176] LNetError: 62936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.706692] LNet: Removed LNI 192.168.203.138@tcp [ 1724.234624] Key type .llcrypt unregistered [ 1724.236641] Key type ._llcrypt unregistered [ 1744.477421] Key type ._llcrypt registered [ 1744.478793] Key type .llcrypt registered [ 1744.553968] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1745.048404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.061457] alg: No test for adler32 (adler32-zlib) [ 1746.107768] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1746.283728] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1747.912165] Key type lgssc registered [ 1748.658256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1754.624503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1754.657386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.105350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.226786] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1765.852793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1766.138873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.966832] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1773.154757] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1775.353245] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1780.083928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1780.279023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.320675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1783.628093] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1787.179623] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1795.912133] Lustre: Failing over lustre-MDT0000 [ 1796.071526] LustreError: 66844:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1796.131969] Lustre: server umount lustre-MDT0000 complete [ 1796.581324] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.582597] LustreError: 65312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.604576] Lustre: Skipped 1 previous similar message [ 1796.614533] LustreError: 65312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1799.835428] LustreError: 66758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.851380] LustreError: 66758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1801.701533] LustreError: 65312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1801.711478] LustreError: 65312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1802.198779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.289476] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.390595] Lustre: *** cfs_fail_loc=135, val=0*** [ 1802.469381] LustreError: 67352:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1802.473054] LustreError: 67352:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -2 [ 1802.481140] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1802.481140] [ 1802.487550] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1802.497516] LustreError: 67340:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1802.506054] LustreError: 67340:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1802.512111] LustreError: 67340:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1802.527464] Lustre: server umount lustre-MDT0000 complete [ 1802.532259] LustreError: 67340:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1804.936645] LustreError: 64593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1806.776547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.847106] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.989768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.017105] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1808.916275] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1810.058797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1812.462320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1812.471545] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1812.494586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1817.058885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.062223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.069963] Lustre: Skipped 1 previous similar message [ 1817.075541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.578189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.579605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.019130] LustreError: 68161:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1820.022510] LustreError: 68161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1820.123057] Lustre: server umount lustre-OST0000 complete [ 1822.692201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1822.692400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.704568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.710119] Lustre: Skipped 2 previous similar messages [ 1827.809383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.823740] Lustre: Skipped 2 previous similar messages [ 1828.765942] LustreError: 68362:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1828.772738] LustreError: 68362:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1828.866190] Lustre: server umount lustre-MDT0000 complete [ 1831.571290] LustreError: 64579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352732 with bad export cookie 17226586927404004662 [ 1831.578118] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.585138] LustreError: 64579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1831.706313] LustreError: 68563:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1831.709946] LustreError: 68563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1831.820293] Lustre: server umount lustre-MDT0001 complete [ 1838.409080] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1839.978562] Key type lgssc unregistered [ 1840.196355] LNet: 69096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.204278] LNetError: 69096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.221653] LNet: Removed LNI 192.168.203.138@tcp [ 1840.693392] Key type .llcrypt unregistered [ 1840.694960] Key type ._llcrypt unregistered [ 1854.354513] Key type ._llcrypt registered [ 1854.357688] Key type .llcrypt registered [ 1854.424375] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1854.881323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1854.926084] alg: No test for adler32 (adler32-zlib) [ 1855.915149] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1856.073055] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1857.691288] Key type lgssc registered [ 1858.327693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.343099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.359341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.671926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.515208] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1873.288434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1873.545943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1875.901336] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1879.234943] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1881.437361] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1885.498975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1885.750785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.978215] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1889.839627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1892.393721] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1900.271306] Lustre: Failing over lustre-MDT0000 [ 1900.415352] LustreError: 72713:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1900.468384] Lustre: server umount lustre-MDT0000 complete [ 1903.079093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.081264] LustreError: 70464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.089300] Lustre: Skipped 2 previous similar messages [ 1903.095300] LustreError: 70464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1904.283071] LustreError: 72482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.297294] LustreError: 72482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1906.178876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.257554] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1906.359195] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1906.427139] LustreError: 73223:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1906.431954] LustreError: 73223:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -9 [ 1906.436617] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1906.436617] [ 1906.442445] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1906.449197] LustreError: 73211:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1906.452843] LustreError: 73211:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1906.457804] LustreError: 73211:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1906.474981] Lustre: server umount lustre-MDT0000 complete [ 1906.476988] LustreError: 73211:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1908.197382] LustreError: 70468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.215626] LustreError: 70468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1910.595853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.686576] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.859609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.901230] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1913.259694] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1914.506674] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1915.883840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1915.897178] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1915.931338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1918.433850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.437375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.444035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.993060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1920.997115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.999489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.826090] LustreError: 74033:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1923.832875] LustreError: 74033:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1923.925098] Lustre: server umount lustre-OST0000 complete [ 1926.114206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.115712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.120258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.126147] Lustre: Skipped 2 previous similar messages [ 1931.233184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.237546] Lustre: Skipped 1 previous similar message [ 1932.182189] LustreError: 74235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1932.185794] LustreError: 74235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1932.258959] Lustre: server umount lustre-MDT0000 complete [ 1934.343380] LustreError: 70448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352835 with bad export cookie 1876721294191569484 [ 1934.344717] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.348989] LustreError: 70448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1934.441267] LustreError: 74437:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1934.444339] LustreError: 74437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1934.533935] Lustre: server umount lustre-MDT0001 complete [ 1941.105616] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1942.908114] Key type lgssc unregistered [ 1943.084635] LNet: 74967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.093171] LNetError: 74967:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.100771] LNet: Removed LNI 192.168.203.138@tcp [ 1943.610850] Key type .llcrypt unregistered [ 1943.614414] Key type ._llcrypt unregistered [ 1949.860780] Key type ._llcrypt registered [ 1949.862293] Key type .llcrypt registered [ 1949.947446] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1950.140775] Key type .llcrypt unregistered [ 1950.142361] Key type ._llcrypt unregistered [ 1963.592176] Key type ._llcrypt registered [ 1963.593958] Key type .llcrypt registered [ 1963.662904] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1964.264706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.273897] alg: No test for adler32 (adler32-zlib) [ 1965.210816] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1965.368920] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1966.984739] Key type lgssc registered [ 1967.691158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.056853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.070765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.340035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.348302] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1980.164931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1980.377919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.483788] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1985.560594] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1987.640745] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1991.274504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1991.470954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.186497] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1997.009470] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1998.634823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2011.109682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.109846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.118194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.123822] Lustre: Skipped 1 previous similar message [ 2016.226264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2016.228660] Lustre: Skipped 1 previous similar message [ 2016.952364] LustreError: 79073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2017.013910] Lustre: server umount lustre-OST0000 complete [ 2021.346700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2021.349874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.352304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.357497] Lustre: Skipped 2 previous similar messages [ 2021.363282] Lustre: Skipped 1 previous similar message [ 2026.466567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.469772] Lustre: Skipped 1 previous similar message [ 2031.585414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.587824] Lustre: Skipped 1 previous similar message [ 2033.120144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2033.166253] LustreError: 79277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2033.169559] LustreError: 79277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2033.210553] Lustre: server umount lustre-MDT0000 complete [ 2034.948913] LustreError: 79074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352936 with bad export cookie 521637442985460027 [ 2034.950880] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2034.956161] LustreError: 79074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2035.050921] LustreError: 79479:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2035.053742] LustreError: 79479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2035.146120] Lustre: server umount lustre-MDT0001 complete [ 2040.375913] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2041.740098] Key type lgssc unregistered [ 2041.889543] LNet: 80010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.893761] LNetError: 80010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.905560] LNet: Removed LNI 192.168.203.138@tcp [ 2042.290495] Key type .llcrypt unregistered [ 2042.292075] Key type ._llcrypt unregistered [ 2054.039719] Key type ._llcrypt registered [ 2054.041154] Key type .llcrypt registered [ 2054.092878] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2054.589526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.603445] alg: No test for adler32 (adler32-zlib) [ 2055.557566] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2055.683796] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2057.296178] Key type lgssc registered [ 2057.847688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2061.531710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2061.543334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.772318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.445714] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2068.348636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2068.532871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.300249] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2072.883170] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2074.520554] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2077.702459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2077.878355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.521845] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2083.097154] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2087.088140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2092.516639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.519602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2092.520122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2092.525846] Lustre: Skipped 1 previous similar message [ 2097.641487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.644888] Lustre: Skipped 1 previous similar message [ 2098.232026] LustreError: 83626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2098.323349] Lustre: server umount lustre-OST0000 complete [ 2102.753788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.755450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.760245] Lustre: Skipped 1 previous similar message [ 2106.642352] LustreError: 83828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2106.645800] LustreError: 83828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2106.701820] Lustre: server umount lustre-MDT0000 complete [ 2107.873225] LustreError: 82501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.881987] LustreError: 82501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2108.552375] LustreError: 81362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353009 with bad export cookie 2700092158507947279 [ 2108.557183] LustreError: 81362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2108.558476] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2108.638865] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2108.641373] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2108.719655] Lustre: server umount lustre-MDT0001 complete [ 2113.421550] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2114.797587] Key type lgssc unregistered [ 2114.973830] LNet: 84562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.978852] LNetError: 84562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.994806] LNet: Removed LNI 192.168.203.138@tcp [ 2115.424297] Key type .llcrypt unregistered [ 2115.426555] Key type ._llcrypt unregistered [ 2127.438196] Key type ._llcrypt registered [ 2127.439447] Key type .llcrypt registered [ 2127.486905] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2127.926052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2127.979538] alg: No test for adler32 (adler32-zlib) [ 2128.859318] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2128.961221] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2130.592477] Key type lgssc registered [ 2131.122926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2134.639386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2134.648090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.881811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.509433] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2140.992643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2141.145530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.716279] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2145.052790] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2146.569276] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2149.706855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.927698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.534420] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2155.030714] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2161.190441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2166.245061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2166.248138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.254718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.362493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.365979] Lustre: Skipped 1 previous similar message [ 2171.567115] LustreError: 88179:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2171.614069] Lustre: server umount lustre-OST0000 complete [ 2176.481573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.489798] Lustre: Skipped 1 previous similar message [ 2176.493068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.496303] Lustre: Skipped 2 previous similar messages [ 2179.264926] LustreError: 88380:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2179.268128] LustreError: 88380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2179.302082] Lustre: server umount lustre-MDT0000 complete [ 2180.880139] LustreError: 85914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353082 with bad export cookie 9715394829722682096 [ 2180.882758] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.885196] LustreError: 85914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2180.960340] LustreError: 88582:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2180.963584] LustreError: 88582:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2181.028654] Lustre: server umount lustre-MDT0001 complete [ 2185.531737] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2186.762262] Key type lgssc unregistered [ 2186.897292] LNet: 89114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.903653] LNetError: 89114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.916402] LNet: Removed LNI 192.168.203.138@tcp [ 2187.292570] Key type .llcrypt unregistered [ 2187.294340] Key type ._llcrypt unregistered [ 2197.876226] Key type ._llcrypt registered [ 2197.877846] Key type .llcrypt registered [ 2197.924663] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2198.307622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.362305] alg: No test for adler32 (adler32-zlib) [ 2199.244848] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2199.341403] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2200.936270] Key type lgssc registered [ 2201.394984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2205.008807] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2205.018629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2220.512332] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2225.632634] LustreError: 90458:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2252.256179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.818473] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2257.248135] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2259.767299] LustreError: 91313:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2259.808647] Lustre: server umount lustre-OST0000 complete [ 2263.464368] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:19:24 (1763353164) [ 2266.275638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.468041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.020814] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2271.714291] LustreError: 92000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.721603] LustreError: 92000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2271.801531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2272.009748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2273.715771] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2276.104648] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2277.718535] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2280.516721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2280.704261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.171978] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2285.714993] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2290.989670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2296.291658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.291776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.293947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2296.296246] Lustre: Skipped 1 previous similar message [ 2301.411357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.417374] Lustre: Skipped 2 previous similar messages [ 2302.201212] LustreError: 94387:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2302.204033] LustreError: 94387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2302.259741] Lustre: server umount lustre-OST0000 complete [ 2306.530372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.531338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.535672] Lustre: Skipped 1 previous similar message [ 2311.649170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.652370] Lustre: Skipped 2 previous similar messages [ 2316.769251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.775740] Lustre: Skipped 1 previous similar message [ 2318.304166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.342057] LustreError: 94588:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2318.344473] LustreError: 94588:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2318.391245] Lustre: server umount lustre-MDT0000 complete [ 2319.958823] LustreError: 94590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353221 with bad export cookie 15479661814026594495 [ 2319.961498] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.964462] LustreError: 94590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2320.111759] Lustre: server umount lustre-MDT0001 complete [ 2324.411065] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2325.555537] Key type lgssc unregistered [ 2325.713611] LNet: 95324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.719484] LNetError: 95324:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.734673] LNet: Removed LNI 192.168.203.138@tcp [ 2326.139365] Key type .llcrypt unregistered [ 2326.149059] Key type ._llcrypt unregistered [ 2337.830989] Key type ._llcrypt registered [ 2337.833297] Key type .llcrypt registered [ 2337.905677] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2338.296159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2338.305731] alg: No test for adler32 (adler32-zlib) [ 2339.237777] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2339.371494] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2340.984176] Key type lgssc registered [ 2341.654639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.825386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.840647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.094522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.069655] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2353.557062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2353.733644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2355.485211] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2357.888516] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2359.558350] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2362.208249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2362.391218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.063783] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2367.536855] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2372.646735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2377.704770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2377.705321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.706488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.719280] Lustre: Skipped 1 previous similar message [ 2378.799924] LustreError: 98943:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2378.851753] Lustre: server umount lustre-OST0000 complete [ 2382.670269] LustreError: 99143:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2382.673116] LustreError: 99143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2382.709477] Lustre: server umount lustre-MDT0000 complete [ 2382.819185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.819741] LustreError: 98854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.825661] Lustre: Skipped 1 previous similar message [ 2382.837283] LustreError: 98854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2384.275692] LustreError: 96674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353285 with bad export cookie 3469268119062688729 [ 2384.278845] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.281895] LustreError: 96674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2384.356658] LustreError: 99344:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2384.359033] LustreError: 99344:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2384.424510] Lustre: server umount lustre-MDT0001 complete [ 2388.620383] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2389.826664] Key type lgssc unregistered [ 2389.973653] LNet: 99875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.976663] LNetError: 99875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2389.986551] LNet: Removed LNI 192.168.203.138@tcp [ 2390.318750] Key type .llcrypt unregistered [ 2390.320499] Key type ._llcrypt unregistered [ 2398.682326] Key type ._llcrypt registered [ 2398.684704] Key type .llcrypt registered [ 2398.730184] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2399.074505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2399.102115] alg: No test for adler32 (adler32-zlib) [ 2399.984327] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2400.096897] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2401.696398] Key type lgssc registered [ 2402.165091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2405.379765] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2405.391510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2420.832316] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2425.952257] LustreError: 101077:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2452.576824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.962962] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2457.175675] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2460.032265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.171804] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2460.178316] LustreError: 101987:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2460.182741] LustreError: 101987:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2460.187590] LustreError: 101987:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2460.227676] Lustre: server umount lustre-MDT0000 complete [ 2460.229790] LustreError: 101987:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2472.930347] LustreError: 102418:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2472.998394] Lustre: server umount lustre-OST0000 complete [ 2477.042974] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2481.080813] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2485.539101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.561292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.262258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2493.856854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.810319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.842416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.959633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2496.978260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2497.020366] Lustre: lustre-MDT0000: new disk, initializing [ 2497.068844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.079442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2498.795235] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2503.980287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2504.011579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2504.063697] Lustre: 104763:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2504.099315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2504.104351] Lustre: Skipped 1 previous similar message [ 2504.145917] Lustre: lustre-MDT0001: new disk, initializing [ 2504.201865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2504.220222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2504.226145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2505.872408] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2509.509428] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2510.893779] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2513.566141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2513.602479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2513.715061] Lustre: lustre-OST0000: new disk, initializing [ 2513.717926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2513.757516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.688586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2514.693616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2514.735583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2515.989158] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2519.688198] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2521.319897] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2521.398877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2522.921483] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2523.017355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2525.154689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2525.155430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.158518] LustreError: Skipped 1 previous similar message [ 2525.164952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.169241] Lustre: Skipped 1 previous similar message [ 2529.845124] LustreError: 107701:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2529.848231] LustreError: 107701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2529.898113] Lustre: server umount lustre-OST0000 complete [ 2534.882485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2534.885665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.891595] Lustre: Skipped 1 previous similar message [ 2534.895863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.752120] LustreError: 107902:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2537.759324] LustreError: 107902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2537.812023] Lustre: server umount lustre-MDT0000 complete [ 2539.758032] LustreError: 104755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353440 with bad export cookie 976212806890091630 [ 2539.761528] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.766991] LustreError: 104755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2539.847973] LustreError: 108103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2539.851323] LustreError: 108103:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2539.954526] Lustre: server umount lustre-MDT0001 complete [ 2543.518166] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:24:04 (1763353444) [ 2544.818281] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2552.222538] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2556.752573] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2561.236556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2564.413501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.285410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2570.510701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2573.494989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2573.529487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.652935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2573.674619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2573.718619] Lustre: lustre-MDT0000: new disk, initializing [ 2573.780159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.790533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2575.702559] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2580.669803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2580.706292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2580.751867] Lustre: 111232:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2580.782279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2580.785247] Lustre: Skipped 1 previous similar message [ 2580.826452] Lustre: lustre-MDT0001: new disk, initializing [ 2580.882683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2580.886938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2582.539186] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2586.214330] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2587.710259] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2590.722247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2590.750200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2590.857534] Lustre: lustre-OST0000: new disk, initializing [ 2590.860193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2590.899276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.902483] Lustre: Skipped 1 previous similar message [ 2593.107688] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2596.739512] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2598.504126] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2599.435251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2599.440731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2599.499416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2599.624213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2601.371948] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2601.470167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2604.513543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.514170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.517266] LustreError: Skipped 1 previous similar message [ 2604.521752] Lustre: Skipped 1 previous similar message [ 2604.522769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.528205] Lustre: Skipped 3 previous similar messages [ 2608.372118] LustreError: 114176:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2608.380063] LustreError: 114176:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2608.434789] Lustre: server umount lustre-OST0000 complete [ 2611.683071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2611.686582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.694808] Lustre: Skipped 1 previous similar message [ 2611.700798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.116795] Lustre: server umount lustre-MDT0000 complete [ 2618.049030] LustreError: 112011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353519 with bad export cookie 976212806890092498 [ 2618.049687] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.056716] LustreError: 112011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2618.138051] LustreError: 114582:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2618.141018] LustreError: 114582:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2618.240446] Lustre: server umount lustre-MDT0001 complete [ 2622.884583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.192972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.481389] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2628.582260] LustreError: 115057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.593569] LustreError: 115057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2630.203027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2632.580386] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2635.764900] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2638.045461] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2642.171376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2646.219810] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2649.147245] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2657.035383] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2658.789264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.792523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.798865] Lustre: Skipped 1 previous similar message [ 2658.802591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.805297] Lustre: Skipped 2 previous similar messages [ 2664.369416] LustreError: 117400:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2664.372626] LustreError: 117400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2664.426323] Lustre: server umount lustre-OST0000 complete [ 2671.073877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.074546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.076612] Lustre: Skipped 3 previous similar messages [ 2671.082758] Lustre: Skipped 2 previous similar messages [ 2672.247516] Lustre: server umount lustre-MDT0000 complete [ 2674.012741] LustreError: 115039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353575 with bad export cookie 976212806890093366 [ 2674.017357] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.018597] LustreError: 115039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2674.192432] Lustre: server umount lustre-MDT0001 complete [ 2678.579234] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 2679.811178] Key type lgssc unregistered [ 2679.961592] LNet: 118335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2679.965207] LNetError: 118335:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2679.975747] LNet: Removed LNI 192.168.203.138@tcp [ 2680.338952] Key type .llcrypt unregistered [ 2680.340366] Key type ._llcrypt unregistered [ 2687.558526] Key type ._llcrypt registered [ 2687.559816] Key type .llcrypt registered [ 2687.609388] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2694.394205] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2694.729765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2694.736829] alg: No test for adler32 (adler32-zlib) [ 2695.618844] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2695.727717] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 2697.320166] Key type lgssc registered [ 2697.769642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2701.007948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.364458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.791740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2708.091466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2710.954165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2710.980033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2710.988850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.093309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2712.107609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2712.149990] Lustre: lustre-MDT0000: new disk, initializing [ 2712.185884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.197184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2713.788495] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2718.357528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2718.384097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2718.420743] Lustre: 121435:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2718.447279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2718.449930] Lustre: Skipped 1 previous similar message [ 2718.487183] Lustre: lustre-MDT0001: new disk, initializing [ 2718.526366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2718.540291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2718.543585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2720.048591] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2723.335779] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2724.799982] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2727.443943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2727.478093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2727.593893] Lustre: lustre-OST0000: new disk, initializing [ 2727.596848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2727.636421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.028675] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2733.418123] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2735.248990] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2737.154316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2737.159245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2737.203536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2737.391388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2738.983220] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2739.075035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2742.242543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.245751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.247546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.250167] Lustre: Skipped 1 previous similar message [ 2745.968109] LustreError: 124387:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2746.015233] Lustre: server umount lustre-OST0000 complete [ 2749.409276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.413335] LustreError: Skipped 1 previous similar message [ 2749.415410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.422409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.425427] Lustre: Skipped 1 previous similar message [ 2752.482604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.483279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.666832] LustreError: 124588:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2753.669241] LustreError: 124588:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2753.701213] Lustre: server umount lustre-MDT0000 complete [ 2755.241126] LustreError: 122216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353656 with bad export cookie 1448049223666493368 [ 2755.244523] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2755.247852] LustreError: 122216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2755.313663] LustreError: 124790:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2755.316502] LustreError: 124790:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2755.387883] Lustre: server umount lustre-MDT0001 complete [ 2758.531885] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:27:39 (1763353659) [ 2761.145892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.350712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.939482] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2766.743913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2766.818824] LustreError: 125460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.827426] LustreError: 125460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2766.934471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.614156] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2770.779181] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2772.181731] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2776.546488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.547392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.553159] Lustre: Skipped 1 previous similar message [ 2776.558637] Lustre: Skipped 2 previous similar messages [ 2782.691392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.694799] Lustre: Skipped 3 previous similar messages [ 2787.296168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2787.340857] LustreError: 126803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2787.345457] LustreError: 126803:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2787.377081] Lustre: server umount lustre-MDT0000 complete [ 2787.810306] LustreError: 125459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.816805] LustreError: 125459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2788.878649] LustreError: 125441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353690 with bad export cookie 1448049223666494236 [ 2788.883404] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.887387] LustreError: 125441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2789.013550] Lustre: server umount lustre-MDT0001 complete [ 2792.207854] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:28:12 (1763353692) [ 2794.808043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2810.272646] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2815.392403] LustreError: 127651:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2840.992308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.051274] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2844.902596] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2845.635437] LustreError: 128313:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2845.637789] LustreError: 128313:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2845.678075] Lustre: server umount lustre-OST0000 complete [ 2848.549615] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:29:09 (1763353749) [ 2851.033484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.207527] LustreError: 128983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.694413] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2855.901610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2857.550850] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2859.518524] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2860.801522] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2863.412504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2863.548960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.551880] Lustre: Skipped 2 previous similar messages [ 2865.692168] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2867.680721] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2882.019260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.020842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.025671] Lustre: Skipped 2 previous similar messages [ 2882.028260] Lustre: Skipped 1 previous similar message [ 2882.032107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2885.886353] LustreError: 131250:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2885.889348] LustreError: 131250:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2885.912849] Lustre: server umount lustre-MDT0000 complete [ 2887.139368] LustreError: 129001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.145049] LustreError: 129001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2887.196436] LustreError: 128964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353788 with bad export cookie 1448049223666494887 [ 2887.200155] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.201207] LustreError: 128964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2887.317088] Lustre: server umount lustre-MDT0001 complete [ 2898.847532] Lustre: server umount lustre-OST0000 complete [ 2901.864184] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:30:02 (1763353802) [ 2904.562297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.744546] LustreError: 132324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.776334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.260336] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2909.241981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2910.687755] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2912.456952] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2913.596816] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.893907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2917.782811] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2919.556421] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2920.869814] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2929.319849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2930.110077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2931.405436] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2931.490429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2934.754458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2934.754905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.758380] LustreError: Skipped 1 previous similar message [ 2934.762970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.766096] Lustre: Skipped 2 previous similar messages [ 2938.222421] LustreError: 135131:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2938.224630] LustreError: 135131:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2938.261237] Lustre: server umount lustre-OST0000 complete [ 2945.695081] Lustre: server umount lustre-MDT0000 complete [ 2946.992827] LustreError: 133044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353848 with bad export cookie 1448049223666495944 [ 2946.995114] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.998438] LustreError: 133044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2947.130877] Lustre: server umount lustre-MDT0001 complete [ 2949.998340] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:30:50 (1763353850) [ 2952.241069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2967.712602] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2972.832540] LustreError: 136180:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 2998.432436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.434940] Lustre: Skipped 2 previous similar messages [ 3000.587477] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3002.691421] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3005.226573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.446185] LustreError: 136207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.537070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3007.110503] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3010.911185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3012.778834] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3015.232926] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3016.776675] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3018.395319] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3018.480766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3019.997085] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3020.091888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3021.281476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3021.284987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.290988] Lustre: Skipped 3 previous similar messages [ 3021.293547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.296215] Lustre: Skipped 4 previous similar messages [ 3035.104115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3035.139486] LustreError: 138960:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3035.142325] LustreError: 138960:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3035.168310] Lustre: server umount lustre-MDT0000 complete [ 3036.488224] LustreError: 136187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353937 with bad export cookie 1448049223666496952 [ 3036.491136] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3036.493182] LustreError: 136187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3036.619441] Lustre: server umount lustre-MDT0001 complete [ 3048.287743] Lustre: server umount lustre-OST0000 complete [ 3051.361128] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:32:32 (1763353952) [ 3053.804089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3069.280551] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3074.400458] LustreError: 140011:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 3101.964968] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3103.948824] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3106.539019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.696637] LustreError: 140038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.706537] LustreError: 140038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3106.754325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3108.163705] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3111.422971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3113.136824] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3115.204784] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3116.550132] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3118.997975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3119.034478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3119.103637] Lustre: lustre-OST0001: new disk, initializing [ 3119.106338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3121.173552] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3124.357960] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3125.720484] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3128.827183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3128.830608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3128.862992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3129.882725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3131.092750] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3131.159580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3132.386225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3132.386323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.390408] LustreError: Skipped 1 previous similar message [ 3132.394658] Lustre: Skipped 2 previous similar messages [ 3132.397658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3132.400808] Lustre: Skipped 10 previous similar messages [ 3146.208105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3146.225511] LustreError: 143775:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3146.227938] LustreError: 143775:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3146.249872] Lustre: server umount lustre-OST0000 complete [ 3149.282299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3149.283054] LustreError: 140028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.285453] LustreError: Skipped 1 previous similar message [ 3149.290261] LustreError: 140028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3153.687135] Lustre: server umount lustre-OST0001 complete [ 3161.178431] Lustre: server umount lustre-MDT0000 complete [ 3162.421105] LustreError: 141646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354063 with bad export cookie 1448049223666497897 [ 3162.425381] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3162.425454] LustreError: 141646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3162.545681] Lustre: server umount lustre-MDT0001 complete [ 3164.405583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3165.715686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3166.950396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3168.193555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3170.295157] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:34:31 (1763354071) [ 3170.845391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3171.351930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3171.837692] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:34:32 (1763354072) [ 3173.955625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.044831] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3174.054703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3174.125655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.128409] Lustre: Skipped 6 previous similar messages [ 3175.311973] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3178.102082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3178.134701] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3178.144685] Lustre: 146013:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.564730] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3181.365147] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3182.615838] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3184.907019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.971968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3186.905139] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3188.244646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3188.702415] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3190.066281] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3190.142697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3191.469895] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3191.544744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3193.313431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3193.316745] LustreError: Skipped 2 previous similar messages [ 3194.259065] Lustre: server umount lustre-OST0000 complete [ 3208.182331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3210.346347] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3212.065336] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3213.479828] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.484929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.485204] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3213.490226] Lustre: Skipped 1 previous similar message [ 3213.623961] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3213.703691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3215.030435] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3215.107349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3216.351364] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff928bc6913000.ost_server_uuid 50 [ 3216.878897] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff928bc6913000.ost_server_uuid in FULL state after 0 sec [ 3216.939035] LustreError: 146019:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.38@tcp) returned error from blocking AST (req@0000000059ab5a82 x1849010260470400 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000000daecd76/0x14188118c9ec02ae lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.38@tcp remote: 0x2dbb8bd12dd407fd expref: 5 pid: 148907 timeout: 3316 lvb_type: 0 [ 3216.947651] LustreError: lustre-MDT0000: A client on nid 192.168.203.38@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3216.950092] LustreError: 146008:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.38@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000000daecd76/0x14188118c9ec02ae lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.38@tcp remote: 0x2dbb8bd12dd407fd expref: 6 pid: 148907 timeout: 0 lvb_type: 0 [ 3218.916241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.921757] Lustre: Skipped 7 previous similar messages [ 3224.014484] Lustre: server umount lustre-OST0000 complete [ 3231.444779] Lustre: server umount lustre-MDT0000 complete [ 3232.734311] LustreError: 146050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354133 with bad export cookie 1448049223666499073 [ 3232.737601] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.738948] LustreError: 146050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3247.072140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3247.166485] Lustre: server umount lustre-MDT0001 complete [ 3250.830206] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3251.872196] Key type lgssc unregistered [ 3252.004634] LNet: 151496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3252.007792] LNetError: 151496:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3252.017427] LNet: Removed LNI 192.168.203.138@tcp [ 3252.344271] Key type .llcrypt unregistered [ 3252.345389] Key type ._llcrypt unregistered [ 3261.905191] Key type ._llcrypt registered [ 3261.906353] Key type .llcrypt registered [ 3261.944170] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3262.296506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3262.335227] alg: No test for adler32 (adler32-zlib) [ 3263.193815] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3263.285203] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3264.864194] Key type lgssc registered [ 3265.247797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3267.935530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3267.944265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.094810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.253208] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3272.885496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3272.996586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.195616] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3275.844318] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.147671] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3279.541948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3279.671612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.646623] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3283.434735] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3293.990548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3301.008310] Lustre: Failing over lustre-MDT0000 [ 3301.198978] LustreError: 155132:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3301.224893] Lustre: server umount lustre-MDT0000 complete [ 3303.904722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3303.906869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.910294] LustreError: 152866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.914277] LustreError: 152866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3304.418699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.418827] LustreError: 152866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.421735] Lustre: Skipped 1 previous similar message [ 3304.429066] LustreError: 152866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3309.538034] LustreError: 152866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.542301] LustreError: 152866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3312.075187] LustreError: 152846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.38@tcp arrived at 1763354213 with bad export cookie 11109893290303650665 [ 3312.081497] LustreError: 152846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3314.503164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.546554] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.629041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.654940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3315.881240] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3316.580329] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3316.583201] Lustre: lustre-MDT0000: Denying connection for new client 3cf609e2-3374-4822-bde3-f35f0c033287 (at 192.168.203.38@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3319.787546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3324.897676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.898661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.899326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.902415] Lustre: Skipped 1 previous similar message [ 3329.394393] LustreError: 156060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3329.396272] LustreError: 156060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3329.432133] Lustre: server umount lustre-OST0000 complete [ 3330.590185] LustreError: 156264:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3330.592487] Lustre: 155632:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3330.595086] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3330.601314] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3330.605113] LustreError: 155632:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bed52b20 x1849010855904384/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3330.610418] LustreError: 155632:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3330.613620] LustreError: 155632:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3330.616400] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 3330.684954] LustreError: 156264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3330.687062] LustreError: 156264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3330.711667] Lustre: server umount lustre-MDT0000 complete [ 3331.908177] LustreError: 155134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354233 with bad export cookie 11109893290303651022 [ 3331.912691] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3331.912939] LustreError: 155134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3331.913301] LustreError: 152865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.043811] LustreError: 156465:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3338.045514] LustreError: 156465:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3338.092868] Lustre: server umount lustre-MDT0001 complete [ 3341.390170] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3342.421053] Key type lgssc unregistered [ 3342.548407] LNet: 156996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3342.550979] LNetError: 156996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3342.559291] LNet: Removed LNI 192.168.203.138@tcp [ 3342.853279] Key type .llcrypt unregistered [ 3342.854810] Key type ._llcrypt unregistered [ 3351.834986] Key type ._llcrypt registered [ 3351.836434] Key type .llcrypt registered [ 3351.874213] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3352.268423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.281794] alg: No test for adler32 (adler32-zlib) [ 3353.143851] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3353.231231] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3354.816160] Key type lgssc registered [ 3355.170747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3357.734965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3357.741369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.896853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.261556] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3363.229851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3363.361969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.807095] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3366.677366] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3367.887639] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3370.159575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3370.284747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.124058] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3373.908731] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3384.615618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3394.017254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3394.020338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.025702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3395.043939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3395.045066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.045366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.046708] Lustre: Skipped 1 previous similar message [ 3396.783123] LustreError: 160632:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3396.818791] Lustre: server umount lustre-OST0000 complete [ 3399.136715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3399.138989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.142708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3404.094473] LustreError: 160836:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3404.096203] LustreError: 160836:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3404.125165] Lustre: server umount lustre-MDT0000 complete [ 3405.282150] LustreError: 159673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.288087] LustreError: 159673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3405.323703] LustreError: 159983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354306 with bad export cookie 6999475155465767562 [ 3405.327088] LustreError: 159983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3405.327608] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.397566] LustreError: 161038:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3405.400099] LustreError: 161038:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3405.445102] Lustre: server umount lustre-MDT0001 complete [ 3408.834700] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3409.848326] Key type lgssc unregistered [ 3409.990837] LNet: 161570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.994499] LNetError: 161570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3410.004573] LNet: Removed LNI 192.168.203.138@tcp [ 3410.329501] Key type .llcrypt unregistered [ 3410.330653] Key type ._llcrypt unregistered [ 3422.066018] Key type ._llcrypt registered [ 3422.067434] Key type .llcrypt registered [ 3422.100359] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3422.489736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3422.496518] alg: No test for adler32 (adler32-zlib) [ 3423.339683] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3423.412564] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3424.984134] Key type lgssc registered [ 3425.311574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3427.734942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3427.740351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.884052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.092963] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3432.821928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3432.939088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3434.168103] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3435.846410] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3436.995301] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3439.116870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3439.239977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.989664] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3442.655757] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3454.632802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3460.067120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.067635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.068408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.071109] Lustre: Skipped 1 previous similar message [ 3461.420147] LustreError: 165498:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3461.454526] Lustre: server umount lustre-OST0000 complete [ 3463.648977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3463.652079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.655259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.656916] Lustre: Skipped 1 previous similar message [ 3464.704558] LustreError: 165699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3464.706395] LustreError: 165699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3464.734258] Lustre: server umount lustre-MDT0000 complete [ 3465.185478] LustreError: 163228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.186297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.190448] LustreError: 163228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3466.021145] LustreError: 164846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354367 with bad export cookie 8600476910894549173 [ 3466.024521] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.025124] LustreError: 164846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3466.095812] LustreError: 165900:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3466.097604] LustreError: 165900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3466.148383] Lustre: server umount lustre-MDT0001 complete [ 3469.816298] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3470.793745] Key type lgssc unregistered [ 3470.929374] LNet: 166431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3470.932989] LNetError: 166431:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3470.941288] LNet: Removed LNI 192.168.203.138@tcp [ 3471.248295] Key type .llcrypt unregistered [ 3471.250102] Key type ._llcrypt unregistered [ 3480.107197] Key type ._llcrypt registered [ 3480.108391] Key type .llcrypt registered [ 3480.152028] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3480.501429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.509568] alg: No test for adler32 (adler32-zlib) [ 3481.364017] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3481.449366] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3483.040157] Key type lgssc registered [ 3483.474187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3487.202956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3487.210393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.367164] Lustre: *** cfs_fail_loc=135, val=0*** [ 3488.418852] LustreError: 167912:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 3488.421471] LustreError: 167912:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -2 [ 3488.425808] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3488.425808] [ 3488.430077] LustreError: MGC192.168.203.138@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3488.434587] LustreError: 167872:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3488.437519] LustreError: 167872:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3488.441350] LustreError: 167872:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 3488.444750] LustreError: 167872:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3488.492577] Lustre: server umount lustre-MDT0000 complete [ 3488.493842] LustreError: 167872:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3490.795408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.952844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.262370] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3495.172941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3495.300600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3496.745172] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3498.663889] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3499.831956] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3500.520575] LustreError: 168313:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3500.525244] LustreError: 168313:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3500.529882] LustreError: 168313:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3505.635348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.635635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3505.641451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.645562] Lustre: Skipped 1 previous similar message [ 3506.687477] LustreError: 169625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3506.689466] LustreError: 169625:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3506.714098] Lustre: server umount lustre-MDT0000 complete [ 3507.909296] LustreError: 168266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354409 with bad export cookie 1730552543496487150 [ 3507.912401] LustreError: 168266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3507.912860] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.044281] LustreError: 169828:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3514.046824] LustreError: 169828:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3514.100079] Lustre: server umount lustre-MDT0001 complete [ 3517.458867] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3518.492510] Key type lgssc unregistered [ 3518.611369] LNet: 170361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.613894] LNetError: 170361:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.624269] LNet: Removed LNI 192.168.203.138@tcp [ 3518.925915] Key type .llcrypt unregistered [ 3518.926997] Key type ._llcrypt unregistered [ 3525.995282] Key type ._llcrypt registered [ 3525.996399] Key type .llcrypt registered [ 3526.036384] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3526.135267] Key type .llcrypt unregistered [ 3526.136602] Key type ._llcrypt unregistered [ 3533.095336] Key type ._llcrypt registered [ 3533.096689] Key type .llcrypt registered [ 3533.138242] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3533.495829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3533.525548] alg: No test for adler32 (adler32-zlib) [ 3534.373909] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3534.466801] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3536.048173] Key type lgssc registered [ 3536.417973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3539.080936] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3539.088756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3554.528402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3559.648212] LustreError: 172429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 3586.272186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.310836] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3590.105810] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3592.294774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.449082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.468355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3593.694400] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3596.407625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3596.531254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.792792] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3599.550539] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.750579] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3601.703602] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3602.912930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3602.915763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.919559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.013555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3607.013563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.013682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.020288] Lustre: Skipped 1 previous similar message [ 3608.689139] LustreError: 174909:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3608.729425] Lustre: server umount lustre-OST0000 complete [ 3612.129476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3612.129802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.130412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3612.137202] Lustre: Skipped 1 previous similar message [ 3615.993339] LustreError: 175110:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3615.995262] LustreError: 175110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3616.022326] Lustre: server umount lustre-MDT0000 complete [ 3617.214048] LustreError: 174752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354518 with bad export cookie 11252188099500755473 [ 3617.217228] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3617.217469] LustreError: 174752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3617.278567] LustreError: 175314:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3617.280626] LustreError: 175314:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3617.325663] Lustre: server umount lustre-MDT0001 complete [ 3620.510337] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3621.466266] Key type lgssc unregistered [ 3621.593685] LNet: 175844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.596534] LNetError: 175844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.606354] LNet: Removed LNI 192.168.203.138@tcp [ 3621.895810] Key type .llcrypt unregistered [ 3621.896736] Key type ._llcrypt unregistered [ 3630.092301] Key type ._llcrypt registered [ 3630.093429] Key type .llcrypt registered [ 3630.127273] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3630.405817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3630.426209] alg: No test for adler32 (adler32-zlib) [ 3631.267860] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3631.348518] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3632.928167] Key type lgssc registered [ 3633.284110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.643858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3635.651751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.783910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.968753] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3640.653554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3640.793860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.099958] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3643.953405] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3645.185363] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.558463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3647.688167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.602967] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3651.421592] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3661.035897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3663.181560] Lustre: Failing over lustre-MDT0000 [ 3663.250563] LustreError: 179530:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3663.281310] Lustre: server umount lustre-MDT0000 complete [ 3666.404368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.405870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3666.407068] LustreError: 177209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.407074] LustreError: 177209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3666.407713] Lustre: Skipped 1 previous similar message [ 3667.085345] LustreError: 177209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.091288] LustreError: 177209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3671.521774] LustreError: 177213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3671.527408] LustreError: 177213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3676.248207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.287218] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3676.376920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.401822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3677.316895] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3677.709325] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3679.281549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3681.767065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3681.777940] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3681.797142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3685.080949] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3697.122454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.123950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.124871] LustreError: Skipped 1 previous similar message [ 3697.128122] Lustre: Skipped 1 previous similar message [ 3697.129682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3697.132165] Lustre: Skipped 1 previous similar message [ 3698.414094] LustreError: 180858:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3698.417029] LustreError: 180858:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3698.450531] Lustre: server umount lustre-OST0000 complete [ 3702.241902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.241956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3702.247597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3702.251750] Lustre: Skipped 1 previous similar message [ 3705.789517] LustreError: 181063:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3705.792410] LustreError: 181063:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3705.824741] Lustre: server umount lustre-MDT0000 complete [ 3707.110740] LustreError: 181064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354608 with bad export cookie 3448726012389938519 [ 3707.113768] LustreError: 181064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3707.113822] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.236189] Lustre: server umount lustre-MDT0001 complete [ 3710.849375] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3711.873724] Key type lgssc unregistered [ 3711.991260] LNet: 181796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3711.993573] LNetError: 181796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3711.999303] LNet: Removed LNI 192.168.203.138@tcp [ 3712.304718] Key type .llcrypt unregistered [ 3712.305980] Key type ._llcrypt unregistered [ 3721.169496] Key type ._llcrypt registered [ 3721.170683] Key type .llcrypt registered [ 3721.210207] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3721.664439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3721.673190] alg: No test for adler32 (adler32-zlib) [ 3722.524175] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3722.614845] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3724.192131] Key type lgssc registered [ 3724.568579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3727.165386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3727.171047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.331535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.626810] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3732.521893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3732.642931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.911418] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3735.638846] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3736.778103] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3739.083055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3739.204390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.110415] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3742.945922] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3754.600647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3759.402527] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3770.262077] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3776.279446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3785.701333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.702741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3785.703818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3785.707438] Lustre: Skipped 1 previous similar message [ 3790.514548] LustreError: 185631:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3790.551661] Lustre: server umount lustre-OST0000 complete [ 3794.400932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3794.403655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.408235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3794.410219] Lustre: Skipped 1 previous similar message [ 3795.938269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3795.938656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.057653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.059432] Lustre: Skipped 2 previous similar messages [ 3806.176090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3806.177680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.180238] Lustre: Skipped 1 previous similar message [ 3806.213507] LustreError: 185832:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3806.216234] LustreError: 185832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3806.249736] Lustre: server umount lustre-MDT0000 complete [ 3807.512674] LustreError: 183147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354708 with bad export cookie 3472740914034005971 [ 3807.518395] LustreError: 183147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3807.518443] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.581543] LustreError: 186034:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3807.583410] LustreError: 186034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3807.629759] Lustre: server umount lustre-MDT0001 complete [ 3810.992167] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3811.991472] Key type lgssc unregistered [ 3812.116355] LNet: 186565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3812.118424] LNetError: 186565:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3812.125310] LNet: Removed LNI 192.168.203.138@tcp [ 3812.398775] Key type .llcrypt unregistered [ 3812.399934] Key type ._llcrypt unregistered [ 3821.540993] Key type ._llcrypt registered [ 3821.542601] Key type .llcrypt registered [ 3821.583772] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3822.150785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.169545] alg: No test for adler32 (adler32-zlib) [ 3823.049791] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3823.157738] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3824.744219] Key type lgssc registered [ 3825.163622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3828.036119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3828.044049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.236788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.646947] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3833.721153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3833.865805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3835.284125] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3837.071213] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3838.288996] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3840.523172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3840.649155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.671500] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3844.592329] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3855.017887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3861.584405] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3870.595726] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3878.776553] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3884.002127] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3891.170758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3891.171854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.173986] LustreError: Skipped 1 previous similar message [ 3891.179293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.182383] Lustre: Skipped 1 previous similar message [ 3895.344610] LustreError: 191931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3895.382644] Lustre: server umount lustre-OST0000 complete [ 3900.384838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3900.387324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.391508] Lustre: Skipped 1 previous similar message [ 3900.393346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.409263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.409654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.781919] LustreError: 192132:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3902.783750] LustreError: 192132:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3902.811862] Lustre: server umount lustre-MDT0000 complete [ 3904.045511] LustreError: 190798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354805 with bad export cookie 5632129020267743101 [ 3904.050693] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3904.050920] LustreError: 190798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3904.122150] LustreError: 192333:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3904.126110] LustreError: 192333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3904.184595] Lustre: server umount lustre-MDT0001 complete [ 3907.937350] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3908.964387] Key type lgssc unregistered [ 3909.096331] LNet: 192863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3909.098624] LNetError: 192863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.106344] LNet: Removed LNI 192.168.203.138@tcp [ 3909.415630] Key type .llcrypt unregistered [ 3909.417659] Key type ._llcrypt unregistered [ 3918.810574] Key type ._llcrypt registered [ 3918.811572] Key type .llcrypt registered [ 3918.850381] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3919.238937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3919.267874] alg: No test for adler32 (adler32-zlib) [ 3920.136946] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3920.224343] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3921.808134] Key type lgssc registered [ 3922.189389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3924.867386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3924.873612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.048637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.412687] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3930.315358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3930.444276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.761095] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3933.543119] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3934.787856] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3937.064579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.201026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.247369] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3941.064571] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3951.529921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3955.680901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.682987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.687809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.706839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3956.706839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.706891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.717782] Lustre: Skipped 1 previous similar message [ 3956.847367] LustreError: 196652:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3956.890403] Lustre: server umount lustre-OST0000 complete [ 3960.253967] LustreError: 196853:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3960.255962] LustreError: 196853:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3960.284685] Lustre: server umount lustre-MDT0000 complete [ 3961.312875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3961.314915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.317980] LustreError: 194252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.322361] LustreError: 194252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3961.506188] LustreError: 194214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354862 with bad export cookie 702105485490793021 [ 3961.509806] LustreError: 194214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3961.510179] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.582688] LustreError: 197055:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3961.584977] LustreError: 197055:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3961.629791] Lustre: server umount lustre-MDT0001 complete [ 3965.125556] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 3966.173357] Key type lgssc unregistered [ 3966.305455] LNet: 197585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3966.308935] LNetError: 197585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3966.316306] LNet: Removed LNI 192.168.203.138@tcp [ 3966.612158] Key type .llcrypt unregistered [ 3966.613322] Key type ._llcrypt unregistered [ 3973.588175] Key type ._llcrypt registered [ 3973.589833] Key type .llcrypt registered [ 3973.628268] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 3974.017766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3974.066315] alg: No test for adler32 (adler32-zlib) [ 3974.927095] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3975.012947] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 3976.592135] Key type lgssc registered [ 3976.946335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3979.582379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3979.588217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.765143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.107567] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3985.047658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3985.167065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.357703] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3988.141473] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3989.347628] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3991.503117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3991.629242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.408595] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 3995.299767] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4004.967991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4010.466607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.466980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.469586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.473803] Lustre: Skipped 1 previous similar message [ 4010.476710] Lustre: Skipped 1 previous similar message [ 4011.438984] LustreError: 201225:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4011.471709] Lustre: server umount lustre-OST0000 complete [ 4015.585762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.591258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.594010] Lustre: Skipped 1 previous similar message [ 4018.815492] LustreError: 201427:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4018.818537] LustreError: 201427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4018.849780] Lustre: server umount lustre-MDT0000 complete [ 4020.105593] LustreError: 198790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354921 with bad export cookie 17311014932758591109 [ 4020.109060] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.109430] LustreError: 198790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4020.174582] LustreError: 201628:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4020.177569] LustreError: 201628:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4020.222832] Lustre: server umount lustre-MDT0001 complete [ 4023.866967] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4024.912056] Key type lgssc unregistered [ 4025.040371] LNet: 202158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4025.043234] LNetError: 202158:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4025.053344] LNet: Removed LNI 192.168.203.138@tcp [ 4025.347487] Key type .llcrypt unregistered [ 4025.348662] Key type ._llcrypt unregistered [ 4032.837826] Key type ._llcrypt registered [ 4032.839219] Key type .llcrypt registered [ 4032.874838] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4033.200287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4033.210631] alg: No test for adler32 (adler32-zlib) [ 4034.082162] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4034.169779] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4035.752166] Key type lgssc registered [ 4036.146457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4038.889233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4038.895638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.054396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.366225] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4044.357961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4044.492308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.939446] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4047.841956] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4049.074947] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4051.403571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4051.552239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.523100] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4055.336163] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4065.892606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4068.512257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4068.536153] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4068.613733] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4070.586701] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4072.450163] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4076.058327] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4086.660134] Lustre: Permanently deactivating lustre-OST0001 [ 4086.667264] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4095.995498] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4101.828501] Lustre: Permanently reactivating lustre-OST0001 [ 4101.833720] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4101.835582] Lustre: Skipped 2 previous similar messages [ 4103.160598] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.168436] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4103.170919] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4103.176977] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4103.191567] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4113.377184] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.377308] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.379345] LustreError: Skipped 1 previous similar message [ 4113.384075] Lustre: Skipped 1 previous similar message [ 4113.385528] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4113.388121] Lustre: Skipped 1 previous similar message [ 4114.914846] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4114.917095] Lustre: Skipped 1 previous similar message [ 4117.036701] LustreError: 206895:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4117.053472] Lustre: server umount lustre-OST0001 complete [ 4120.034804] LustreError: 204983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.036111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.037166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.041663] LustreError: 204983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4120.045216] Lustre: Skipped 1 previous similar message [ 4124.333120] LustreError: 207097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4124.334724] LustreError: 207097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4124.371672] Lustre: server umount lustre-OST0000 complete [ 4126.688792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4126.691289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.695333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4126.697231] Lustre: Skipped 1 previous similar message [ 4131.647597] LustreError: 207298:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4131.649314] LustreError: 207298:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4131.679958] Lustre: server umount lustre-MDT0000 complete [ 4132.884556] LustreError: 205006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355034 with bad export cookie 2815140005382152238 [ 4132.888136] LustreError: 205006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4132.889479] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.026101] Lustre: server umount lustre-MDT0001 complete [ 4136.394656] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4137.481742] Key type lgssc unregistered [ 4137.597329] LNet: 208031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4137.600603] LNetError: 208031:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4137.613301] LNet: Removed LNI 192.168.203.138@tcp [ 4137.914348] Key type .llcrypt unregistered [ 4137.915489] Key type ._llcrypt unregistered [ 4148.601494] Key type ._llcrypt registered [ 4148.602989] Key type .llcrypt registered [ 4148.644370] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 4154.591843] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4155.128903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4155.141948] alg: No test for adler32 (adler32-zlib) [ 4156.006128] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4156.092484] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4157.672141] Key type lgssc registered [ 4158.028133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4160.777148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4162.709262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4164.579452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.472074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4168.718426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4168.737531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4168.742262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.840823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4169.856161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4169.898624] Lustre: lustre-MDT0000: new disk, initializing [ 4169.924657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.931100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4171.277686] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4175.500462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4175.528843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4175.564537] Lustre: 211948:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4175.589109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4175.590951] Lustre: Skipped 1 previous similar message [ 4175.629166] Lustre: lustre-MDT0001: new disk, initializing [ 4175.662995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4175.675403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4175.679055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4176.992924] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4180.099894] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4181.353785] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4183.561063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.583240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4183.676191] Lustre: lustre-OST0000: new disk, initializing [ 4183.677942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4183.707449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.601125] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4187.578671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.583025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4187.617682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4188.747947] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4190.077493] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4190.140936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4191.463589] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4191.544056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4192.737925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4192.738134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.740965] LustreError: Skipped 1 previous similar message [ 4192.746793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4192.750458] Lustre: Skipped 1 previous similar message [ 4194.221790] LustreError: 214888:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4194.259164] Lustre: server umount lustre-OST0000 complete [ 4196.321109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4196.324089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.329452] Lustre: Skipped 1 previous similar message [ 4196.331185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.857186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.857918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.598416] LustreError: 215088:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4201.600244] LustreError: 215088:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4201.629152] Lustre: server umount lustre-MDT0000 complete [ 4202.897407] LustreError: 212729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355104 with bad export cookie 1984246507064975596 [ 4202.900736] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.902932] LustreError: 212729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4202.964769] LustreError: 215291:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4202.966924] LustreError: 215291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4203.021781] Lustre: server umount lustre-MDT0001 complete [ 4205.716682] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 23:51:46 (1763355106) [ 4207.961683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.143096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.449870] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4212.306145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4212.425909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4213.783730] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4215.547679] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4216.790642] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4219.041064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4220.989544] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4222.687100] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4239.106107] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4248.688248] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4254.201843] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4262.784149] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4277.875219] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4277.877733] Lustre: Skipped 1 previous similar message [ 4296.434022] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4296.436774] Lustre: Skipped 2 previous similar messages [ 4328.989134] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4328.991336] Lustre: Skipped 4 previous similar messages [ 4391.394586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4391.394966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.397576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4391.400320] Lustre: Skipped 1 previous similar message [ 4391.404358] Lustre: Skipped 2 previous similar messages [ 4396.514162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4396.515704] Lustre: Skipped 1 previous similar message [ 4396.652086] LustreError: 219435:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4396.653662] LustreError: 219435:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4396.685363] Lustre: server umount lustre-OST0000 complete [ 4401.633402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.753144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4406.755155] Lustre: Skipped 3 previous similar messages [ 4412.384118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4412.421042] LustreError: 219641:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4412.423092] LustreError: 219641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4412.449272] Lustre: server umount lustre-MDT0000 complete [ 4413.671311] LustreError: 215941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355314 with bad export cookie 1984246507064976464 [ 4413.674156] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4413.677305] LustreError: 215941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4413.795570] Lustre: server umount lustre-MDT0001 complete [ 4417.342271] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4418.484184] Key type lgssc unregistered [ 4418.615383] LNet: 220374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.617671] LNetError: 220374:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4418.625343] LNet: Removed LNI 192.168.203.138@tcp [ 4418.912867] Key type .llcrypt unregistered [ 4418.914536] Key type ._llcrypt unregistered [ 4427.558586] Key type ._llcrypt registered [ 4427.559513] Key type .llcrypt registered [ 4427.602603] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4428.006452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4428.049711] alg: No test for adler32 (adler32-zlib) [ 4428.917956] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4429.013359] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4430.600179] Key type lgssc registered [ 4430.988339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4433.780902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4433.789842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.965532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.382908] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4439.343487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4439.480813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4440.871881] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4442.770484] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.089752] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.398345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4446.517327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.376132] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4450.143888] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4480.481351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4480.483401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.488063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4481.506937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4481.508623] Lustre: Skipped 2 previous similar messages [ 4484.846401] LustreError: 224189:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4484.882111] Lustre: server umount lustre-OST0000 complete [ 4486.625568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.626382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4486.629683] Lustre: Skipped 2 previous similar messages [ 4492.221194] LustreError: 224394:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4492.223842] LustreError: 224394:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4492.249679] Lustre: server umount lustre-MDT0000 complete [ 4492.769850] LustreError: 222470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.774607] LustreError: 222470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4493.469272] LustreError: 221724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355394 with bad export cookie 863197554301964429 [ 4493.473335] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4493.473483] LustreError: 221724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4493.546532] LustreError: 224597:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4493.548746] LustreError: 224597:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4493.600675] Lustre: server umount lustre-MDT0001 complete [ 4497.044385] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4498.044997] Key type lgssc unregistered [ 4498.174391] LNet: 225127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4498.176851] LNetError: 225127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4498.187319] LNet: Removed LNI 192.168.203.138@tcp [ 4498.519587] Key type .llcrypt unregistered [ 4498.521309] Key type ._llcrypt unregistered [ 4505.781562] Key type ._llcrypt registered [ 4505.782802] Key type .llcrypt registered [ 4505.825288] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4505.932384] Key type .llcrypt unregistered [ 4505.933456] Key type ._llcrypt unregistered [ 4514.186784] Key type ._llcrypt registered [ 4514.188451] Key type .llcrypt registered [ 4514.239667] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 4520.859681] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4521.237082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4521.284651] alg: No test for adler32 (adler32-zlib) [ 4522.157553] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4522.256916] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4523.840191] Key type lgssc registered [ 4524.227159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4527.372883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4529.592963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4531.658155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.764570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4534.266796] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4278 sec ========== 23:57:15 (1763355435) [ 4534.849428] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:57:15 (1763355435) === [ 4536.044746] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:57:16 (1763355436) === [ 4548.634676] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4549.571725] Key type lgssc unregistered [ 4549.685320] LNet: 231204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4549.687876] LNetError: 231204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4549.694244] LNet: Removed LNI 192.168.203.138@tcp [ 4549.955792] Key type .llcrypt unregistered [ 4549.956979] Key type ._llcrypt unregistered