[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472794664 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002369] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007596] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011067] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.022777] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027027] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.032351] *** VALIDATE proc *** [ 0.033216] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039021] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043368] debug: unmapping init [mem 0xffffffff8aa59000-0xffffffff8aa60fff] [ 0.045929] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047016] ... version: 2 [ 0.047891] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052020] ... event mask: 000000070000000f [ 0.053303] rcu: Hierarchical SRCU implementation. [ 0.055417] smp: Bringing up secondary CPUs ... [ 0.056578] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.060373] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.219072] node 0 deferred pages initialised in 154ms [ 0.222390] devtmpfs: initialized [ 0.223298] x86/mm: Memory block size: 128MB [ 0.226875] gcov: version magic: 0x41383552 [ 0.230091] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.231081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233318] pinctrl core: initialized pinctrl subsystem [ 0.235206] [ 0.235740] ************************************************************* [ 0.238013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240008] ** ** [ 0.242011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244011] ** ** [ 0.246013] ** This means that this kernel is built to expose internal ** [ 0.247011] ** IOMMU data structures, which may compromise security on ** [ 0.249013] ** your system. ** [ 0.251014] ** ** [ 0.253014] ** If you see this message and you are not debugging the ** [ 0.255012] ** kernel, report this immediately to your vendor! ** [ 0.257013] ** ** [ 0.259013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261014] ************************************************************* [ 0.263644] NET: Registered protocol family 16 [ 0.265437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.273048] cpuidle: using governor menu [ 0.274803] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.276597] PCI: Using configuration type 1 for base access [ 0.279139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288178] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.294083] cryptd: max_cpu_qlen set to 1000 [ 0.296277] ACPI: Added _OSI(Module Device) [ 0.297023] ACPI: Added _OSI(Processor Device) [ 0.299011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.300012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306051] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.312487] ACPI: Interpreter enabled [ 0.314061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.315012] ACPI: Using IOAPIC for interrupt routing [ 0.317228] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.331294] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.333041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.335017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.342336] acpiphp: Slot [2] registered [ 0.344159] acpiphp: Slot [5] registered [ 0.346109] acpiphp: Slot [6] registered [ 0.347111] acpiphp: Slot [7] registered [ 0.348143] acpiphp: Slot [8] registered [ 0.350106] acpiphp: Slot [9] registered [ 0.351070] acpiphp: Slot [10] registered [ 0.352042] acpiphp: Slot [3] registered [ 0.353122] acpiphp: Slot [4] registered [ 0.354094] acpiphp: Slot [11] registered [ 0.356117] acpiphp: Slot [12] registered [ 0.357109] acpiphp: Slot [13] registered [ 0.359074] acpiphp: Slot [14] registered [ 0.360109] acpiphp: Slot [15] registered [ 0.362126] acpiphp: Slot [16] registered [ 0.363089] acpiphp: Slot [17] registered [ 0.364095] acpiphp: Slot [18] registered [ 0.366094] acpiphp: Slot [19] registered [ 0.367100] acpiphp: Slot [20] registered [ 0.369093] acpiphp: Slot [21] registered [ 0.370097] acpiphp: Slot [22] registered [ 0.372113] acpiphp: Slot [23] registered [ 0.373088] acpiphp: Slot [24] registered [ 0.375092] acpiphp: Slot [25] registered [ 0.376100] acpiphp: Slot [26] registered [ 0.377145] acpiphp: Slot [27] registered [ 0.379096] acpiphp: Slot [28] registered [ 0.380140] acpiphp: Slot [29] registered [ 0.382108] acpiphp: Slot [30] registered [ 0.383099] acpiphp: Slot [31] registered [ 0.384086] PCI host bridge to bus 0000:00 [ 0.386019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.388021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.390034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.392023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.394022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.396031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.398508] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.400872] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.404289] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413847] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.418064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.421020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.423021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.426028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428579] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.431739] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.433034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.434556] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.438012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.453019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456977] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.467016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.479015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.488178] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.496020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533451] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.549017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.595017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.605077] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.613018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.620018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.650891] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.657019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.678019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.688332] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.698018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.724019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.736160] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.738476] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.740359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.742345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.745202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.750044] iommu: Default domain type: Passthrough [ 0.751444] SCSI subsystem initialized [ 0.753148] ACPI: bus type USB registered [ 0.755120] usbcore: registered new interface driver usbfs [ 0.757090] usbcore: registered new interface driver hub [ 0.760102] usbcore: registered new device driver usb [ 0.762195] pps_core: LinuxPPS API ver. 1 registered [ 0.763007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.766061] PTP clock support registered [ 0.768225] EDAC MC: Ver: 3.0.0 [ 0.770122] PCI: Using ACPI for IRQ routing [ 0.771530] NetLabel: Initializing [ 0.773021] NetLabel: domain hash size = 128 [ 0.775022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.777119] NetLabel: unlabeled traffic allowed by default [ 0.780175] vgaarb: loaded [ 0.781332] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.783013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.789360] clocksource: Switched to clocksource kvm-clock [ 0.900214] VFS: Disk quotas dquot_6.6.0 [ 0.901968] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.904696] *** VALIDATE ramfs *** [ 0.906193] *** VALIDATE hugetlbfs *** [ 0.908095] pnp: PnP ACPI init [ 0.911427] pnp: PnP ACPI: found 6 devices [ 0.931526] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.934603] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937053] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939222] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.941323] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.943548] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.946572] NET: Registered protocol family 2 [ 0.949204] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.953943] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957171] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.962362] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.965383] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968321] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971361] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974060] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.976959] NET: Registered protocol family 1 [ 0.979213] RPC: Registered named UNIX socket transport module. [ 0.981354] RPC: Registered udp transport module. [ 0.983014] RPC: Registered tcp transport module. [ 0.984622] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986700] NET: Registered protocol family 44 [ 0.988153] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990075] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.992138] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.994214] PCI: CLS 0 bytes, default 64 [ 0.995843] Unpacking initramfs... [ 2.369892] debug: unmapping init [mem 0xffff9b567cc54000-0xffff9b567ffbffff] [ 2.372743] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.374667] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.377402] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.856527] Initialise system trusted keyrings [ 2.857738] Key type blacklist registered [ 2.859063] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.865635] zbud: loaded [ 2.868142] *** VALIDATE nfs *** [ 2.868983] *** VALIDATE nfs4 *** [ 2.870206] pstore: using deflate compression [ 2.874763] Platform Keyring initialized [ 2.981223] NET: Registered protocol family 38 [ 2.982386] Key type asymmetric registered [ 2.983819] Asymmetric key parser 'x509' registered [ 2.985176] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.986847] io scheduler mq-deadline registered [ 2.988149] io scheduler kyber registered [ 2.989232] io scheduler bfq registered [ 2.991036] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.993646] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.996515] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.999361] ACPI: Power Button [PWRF] [ 3.003413] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.007893] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.015939] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.021658] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.031950] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.063511] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.091954] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.096395] Non-volatile memory driver v1.3 [ 3.097864] Linux agpgart interface v0.103 [ 3.128910] virtio_blk virtio1: [vda] 134104 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.131597] vda: detected capacity change from 0 to 68661248 [ 3.149482] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.152560] vdb: detected capacity change from 0 to 1073741824 [ 3.169325] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.172844] vdc: detected capacity change from 0 to 2621440000 [ 3.188469] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.191699] vdd: detected capacity change from 0 to 2621440000 [ 3.204967] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.208030] vde: detected capacity change from 0 to 4294967296 [ 3.221962] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.224768] vdf: detected capacity change from 0 to 4294967296 [ 3.230091] libphy: Fixed MDIO Bus: probed [ 3.241815] usbcore: registered new interface driver usbserial_generic [ 3.244116] usbserial: USB Serial support registered for generic [ 3.246360] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.250903] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.252440] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.254453] mousedev: PS/2 mouse device common for all mice [ 3.259193] rtc_cmos 00:05: RTC can wake from S4 [ 3.261785] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.264293] rtc_cmos 00:05: registered as rtc0 [ 3.267156] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.268252] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.270099] intel_pstate: CPU model not supported [ 3.276203] hid: raw HID events driver (C) Jiri Kosina [ 3.278630] usbcore: registered new interface driver usbhid [ 3.279084] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.281179] usbhid: USB HID core driver [ 3.285668] drop_monitor: Initializing network drop monitor service [ 3.288582] Initializing XFRM netlink socket [ 3.291188] NET: Registered protocol family 10 [ 3.295064] Segment Routing with IPv6 [ 3.296378] NET: Registered protocol family 17 [ 3.298739] mpls_gso: MPLS GSO support [ 3.304888] RAS: Correctable Errors collector initialized. [ 3.306745] AVX version of gcm_enc/dec engaged. [ 3.308127] AES CTR mode by8 optimization enabled [ 3.380674] sched_clock: Marking stable (3380653807, 0)->(4212492283, -831838476) [ 3.384290] registered taskstats version 1 [ 3.386499] Loading compiled-in X.509 certificates [ 3.388386] zswap: loaded using pool lzo/zbud [ 3.410335] Key type big_key registered [ 3.423173] Key type encrypted registered [ 3.424563] ima: No TPM chip found, activating TPM-bypass! [ 3.426366] ima: Allocated hash algorithm: sha1 [ 3.428305] ima: No architecture policies found [ 3.429821] evm: Initialising EVM extended attributes: [ 3.431550] evm: security.selinux [ 3.436549] evm: security.ima [ 3.437396] evm: security.capability [ 3.438486] evm: HMAC attrs: 0x1 [ 3.440388] rtc_cmos 00:05: setting system clock to 2026-01-16 08:59:27 UTC (1768553967) [ 3.444947] debug: unmapping init [mem 0xffffffff8ba03000-0xffffffff8bbfffff] [ 3.447174] debug: unmapping init [mem 0xffffffff8a782000-0xffffffff8aa58fff] [ 3.456069] Write protecting the kernel read-only data: 28672k [ 3.459113] debug: unmapping init [mem 0xffffffff88e03000-0xffffffff88ffffff] [ 3.461715] debug: unmapping init [mem 0xffffffff89714000-0xffffffff897fffff] [ 3.493962] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.498273] systemd[1]: Detected virtualization kvm. [ 3.499329] systemd[1]: Detected architecture x86-64. [ 3.500587] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.522560] systemd[1]: No hostname configured. [ 3.524365] systemd[1]: Set hostname to . [ 3.526145] random: systemd: uninitialized urandom read (16 bytes read) [ 3.527571] systemd[1]: Initializing machine ID from random generator. [ 3.638976] random: systemd: uninitialized urandom read (16 bytes read) [ 3.641220] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.644675] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646102] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.650320] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.208694] device-mapper: uevent: version 1.0.3 [ 4.210790] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.888315] virtio_net virtio0 ens2: renamed from eth0 [ 4.909181] random: fast init done [ 4.912362] scsi host0: ata_piix [ 4.967741] scsi host1: ata_piix [ 4.969847] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.971814] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.508604] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.943826] random: crng init done [ 9.945463] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.395982] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.623394] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.874961] SELinux: Disabled at runtime. [ 11.940674] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.948859] systemd[1]: Detected virtualization kvm. [ 11.950687] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.414415] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.417976] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.422975] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.427016] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.430425] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.441225] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.451037] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.554814] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.926657] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.174693] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.318707] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.470465] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.479800] EDAC sbridge: Ver: 1.1.2 [ 15.133182] Key type dns_resolver registered [ 15.427318] NFS: Registering the id_resolver key type [ 15.429227] Key type id_resolver registered [ 15.430575] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg233-server login: [ 41.309447] libcfs: loading out-of-tree module taints kernel. [ 41.330664] Key type ._llcrypt registered [ 41.332443] Key type .llcrypt registered [ 41.383495] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 49.784728] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 50.953575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.994687] alg: No test for adler32 (adler32-zlib) [ 52.412973] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 53.265826] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 55.079312] Key type lgssc registered [ 57.051509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.973892] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.325417] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 127.063050] hrtimer: interrupt took 5322563 ns [ 135.599599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.694758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.157715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.215678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.352215] Lustre: lustre-MDT0000: new disk, initializing [ 137.473307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.491724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.003718] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.656484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 155.850376] Lustre: 6501:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.891234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.899285] Lustre: Skipped 1 previous similar message [ 156.038910] Lustre: lustre-MDT0001: new disk, initializing [ 156.137700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.173430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.203392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.763124] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.451422] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.802857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.127373] Lustre: lustre-OST0000: new disk, initializing [ 176.141708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.255637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.905645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 178.925520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.078207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 183.125685] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.305893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 198.432390] Lustre: lustre-OST0001: new disk, initializing [ 198.435803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 198.498401] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.522794] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.443648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.455050] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.519071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 217.026741] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.671632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.756844] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.995510] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 244.308977] Lustre: DEBUG MARKER: Client: 2.17.50.30 [ 246.916605] Lustre: DEBUG MARKER: MDS: 2.17.50.30 [ 250.185147] Lustre: DEBUG MARKER: OSS: 2.17.50.30 [ 252.213420] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 16 04:03:34 EST 2026 [ 270.438600] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 272.135144] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.248210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 289.257842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.270918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.762499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.775176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.786136] Lustre: Skipped 2 previous similar messages [ 294.543569] LustreError: 12196:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 294.781869] Lustre: server umount lustre-MDT0000 complete [ 298.770815] LustreError: 6493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554262 with bad export cookie 16843374839785810182 [ 298.771148] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.785840] LustreError: 6493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 300.000170] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.001752] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.005217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.010991] Lustre: Skipped 3 previous similar messages [ 300.041728] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 304.032231] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.032468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.054412] Lustre: Skipped 2 previous similar messages [ 308.128914] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.129469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.139042] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 308.167512] Lustre: Skipped 1 previous similar message [ 313.313345] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 313.316477] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.352506] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 313.466090] LustreError: 12398:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 313.473808] LustreError: 12398:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 313.694044] Lustre: server umount lustre-MDT0001 complete [ 318.428520] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 318.438142] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 318.622359] Lustre: server umount lustre-OST0000 complete [ 322.470666] LustreError: 12802:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 322.481926] LustreError: 12802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 322.768957] Lustre: server umount lustre-OST0001 complete [ 330.971782] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 340.984511] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 379.237830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.504593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.548501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 379.604995] Lustre: lustre-MDT0000: new disk, initializing [ 379.678915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.696378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 383.607236] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 393.471240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 393.566577] Lustre: 14979:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 393.658352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 393.667505] Lustre: Skipped 1 previous similar message [ 393.767728] Lustre: lustre-MDT0001: new disk, initializing [ 393.890745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.937420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 393.949464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 397.234786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 404.514695] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 408.388890] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 414.505090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 414.762171] Lustre: lustre-OST0000: new disk, initializing [ 414.766446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.851446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.258248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.263566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.354243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 420.473594] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 428.400987] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 433.608441] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 433.868786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 439.127468] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 439.383953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.824357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.824512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.837732] LustreError: Skipped 1 previous similar message [ 441.843124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.843138] Lustre: Skipped 2 previous similar messages [ 441.878865] Lustre: Skipped 1 previous similar message [ 447.513240] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 447.518556] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 447.681626] Lustre: server umount lustre-OST0000 complete [ 452.070443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.077860] Lustre: Skipped 1 previous similar message [ 457.735331] LustreError: 18111:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 457.748209] LustreError: 18111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 458.136650] Lustre: server umount lustre-MDT0000 complete [ 462.304711] LustreError: 14989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.337761] LustreError: 14989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 463.054340] LustreError: 16718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554427 with bad export cookie 16843374839785811715 [ 463.064119] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.078062] LustreError: 16718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 463.923205] Lustre: server umount lustre-MDT0001 complete [ 468.701457] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:07:10 (1768554430) [ 476.856338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.527944] LustreError: 18839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.662171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.230315] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 490.729166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 495.861633] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 503.071550] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 507.582565] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 515.608041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 521.685459] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 527.149930] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 538.085867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 538.087541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.098873] LustreError: Skipped 1 previous similar message [ 538.104928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 538.105419] Lustre: Skipped 1 previous similar message [ 538.124502] Lustre: Skipped 8 previous similar messages [ 540.297504] LustreError: 21078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 540.304147] LustreError: 21078:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 540.463655] Lustre: server umount lustre-OST0000 complete [ 547.808866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.348381] Lustre: server umount lustre-MDT0000 complete [ 553.441954] LustreError: 18839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.459084] LustreError: 18839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 554.052183] LustreError: 18818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554518 with bad export cookie 16843374839785812597 [ 554.056123] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.064555] LustreError: 18818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 554.408455] Lustre: server umount lustre-MDT0001 complete [ 564.221395] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 567.271450] Key type lgssc unregistered [ 567.594615] LNet: 22019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.603962] LNetError: 22019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.652729] LNet: Removed LNI 192.168.202.133@tcp [ 569.027372] Key type .llcrypt unregistered [ 569.031940] Key type ._llcrypt unregistered [ 597.763598] Key type ._llcrypt registered [ 597.766782] Key type .llcrypt registered [ 597.920806] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 599.164607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 599.372543] alg: No test for adler32 (adler32-zlib) [ 600.518396] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 600.776371] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 602.559230] Key type lgssc registered [ 604.303746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 614.147811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 614.193992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.084920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.867889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 626.657714] LustreError: 23413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.668608] LustreError: 23413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 631.780221] LustreError: 23413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.127157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 632.792705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 638.424163] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 646.130399] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 651.162291] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 659.186443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 659.743384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.818881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 666.765873] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 674.969688] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 695.356432] LustreError: 26102:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 695.462657] Lustre: server umount lustre-OST0000 complete [ 702.952130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.968395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.992122] Lustre: Skipped 1 previous similar message [ 708.082156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.096066] Lustre: Skipped 1 previous similar message [ 710.218413] LustreError: 26303:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 710.230837] LustreError: 26303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 710.431685] Lustre: server umount lustre-MDT0000 complete [ 713.188825] LustreError: 23413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.204837] Lustre: 22726:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768554661/real 1768554661] req@00000000ca23d2f7 x1854463870532352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768554677 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 713.213567] LustreError: 23413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 713.257180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.781354] LustreError: 23393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554678 with bad export cookie 15944916923563953248 [ 714.786546] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.805377] LustreError: 23393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 715.070300] LustreError: 26505:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 715.080610] LustreError: 26505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 715.441348] Lustre: server umount lustre-MDT0001 complete [ 727.596995] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 730.359275] Key type lgssc unregistered [ 730.671437] LNet: 27039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.680703] LNetError: 27039:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.701823] LNet: Removed LNI 192.168.202.133@tcp [ 731.869150] Key type .llcrypt unregistered [ 731.870900] Key type ._llcrypt unregistered [ 756.743761] Key type ._llcrypt registered [ 756.746327] Key type .llcrypt registered [ 756.855762] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 758.014297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.039184] alg: No test for adler32 (adler32-zlib) [ 759.008833] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 759.190507] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 760.897329] Key type lgssc registered [ 762.461953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 771.530539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 771.555113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.169704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.606460] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 783.341724] LustreError: 28434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 783.356127] LustreError: 28434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 786.839179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 787.256191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 792.838932] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 800.013159] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 805.815778] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 821.629634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 822.052588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.862396] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 835.382858] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 843.570492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 854.002040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.017279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 854.019912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 854.025268] Lustre: Skipped 1 previous similar message [ 855.351334] LustreError: 31045:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 855.514519] Lustre: server umount lustre-OST0000 complete [ 859.104618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.112754] Lustre: Skipped 1 previous similar message [ 859.113235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 859.116051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.225916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.241285] Lustre: Skipped 2 previous similar messages [ 865.346826] LustreError: 31246:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 865.351530] LustreError: 31246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 865.428301] Lustre: server umount lustre-MDT0000 complete [ 869.344087] LustreError: 29365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.376756] LustreError: 29365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 869.705267] LustreError: 28414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554833 with bad export cookie 2232503572354891996 [ 869.721476] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.723440] LustreError: 28414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 869.883727] LustreError: 31448:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 869.895625] LustreError: 31448:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 870.228432] Lustre: server umount lustre-MDT0001 complete [ 882.762552] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 886.239632] Key type lgssc unregistered [ 886.629990] LNet: 31982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.645684] LNetError: 31982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.660833] LNet: Removed LNI 192.168.202.133@tcp [ 887.376473] Key type .llcrypt unregistered [ 887.377713] Key type ._llcrypt unregistered [ 914.291242] Key type ._llcrypt registered [ 914.294461] Key type .llcrypt registered [ 914.432472] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 915.644550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.944398] alg: No test for adler32 (adler32-zlib) [ 917.081121] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 917.369736] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 919.031170] Key type lgssc registered [ 920.456415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 929.258429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 929.286010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.937587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.821786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 940.518558] LustreError: 33379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 940.535281] LustreError: 33379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 945.641581] LustreError: 33379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.740194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 946.197785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 951.478452] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 958.193804] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.921778] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 969.222088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 969.528973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.666987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 975.749354] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 982.249294] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 992.231858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.236945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.259956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.320611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 996.327497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.327808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.352318] Lustre: Skipped 1 previous similar message [ 997.616218] LustreError: 35622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 997.827263] Lustre: server umount lustre-OST0000 complete [ 1002.465530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1002.478678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.492165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.567439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.581803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.611339] Lustre: Skipped 1 previous similar message [ 1007.747090] LustreError: 35823:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1007.757037] LustreError: 35823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1007.901419] Lustre: server umount lustre-MDT0000 complete [ 1011.684419] LustreError: 33374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.703392] LustreError: 33374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1012.223052] LustreError: 33359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768554976 with bad export cookie 13333568577923903204 [ 1012.230296] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.234964] LustreError: 33359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1012.379702] LustreError: 36025:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1012.384693] LustreError: 36025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1012.514437] Lustre: server umount lustre-MDT0001 complete [ 1025.431457] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1028.693187] Key type lgssc unregistered [ 1029.085340] LNet: 36558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.094685] LNetError: 36558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.118037] LNet: Removed LNI 192.168.202.133@tcp [ 1030.132375] Key type .llcrypt unregistered [ 1030.135413] Key type ._llcrypt unregistered [ 1053.761931] Key type ._llcrypt registered [ 1053.764500] Key type .llcrypt registered [ 1053.844778] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1054.735387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.790325] alg: No test for adler32 (adler32-zlib) [ 1056.002616] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1056.186684] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1057.855173] Key type lgssc registered [ 1058.934828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.909913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1066.938650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.592887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.288661] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1078.753170] LustreError: 37952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.770670] LustreError: 37952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1082.849959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1083.281953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.951618] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1094.025908] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1098.529608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1106.626112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1106.963954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.210519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1113.439409] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1119.226826] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.113171] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1128.428405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.440452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.462560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.928789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.942575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.204049] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1132.210850] Lustre: Skipped 1 previous similar message [ 1132.348189] LustreError: 40197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1132.615515] Lustre: server umount lustre-OST0000 complete [ 1154.528713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.556450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.567124] Lustre: Skipped 1 previous similar message [ 1159.657959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.665645] Lustre: Skipped 1 previous similar message [ 1168.351277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1168.458059] LustreError: 40575:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1168.478595] LustreError: 40575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1168.661635] Lustre: server umount lustre-MDT0000 complete [ 1169.895440] LustreError: 40067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.917373] LustreError: 40067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1173.675405] LustreError: 37934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555137 with bad export cookie 18359951563671857421 [ 1173.684114] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.688889] LustreError: 37934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1180.013860] LustreError: 40778:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1180.022362] LustreError: 40778:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1180.283303] Lustre: server umount lustre-MDT0001 complete [ 1191.921149] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1195.640392] Key type lgssc unregistered [ 1196.032780] LNet: 41311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.042471] LNetError: 41311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.062672] LNet: Removed LNI 192.168.202.133@tcp [ 1197.264703] Key type .llcrypt unregistered [ 1197.268978] Key type ._llcrypt unregistered [ 1225.208476] Key type ._llcrypt registered [ 1225.210095] Key type .llcrypt registered [ 1225.366396] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1226.623135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1226.669129] alg: No test for adler32 (adler32-zlib) [ 1227.774719] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1227.986117] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1229.639182] Key type lgssc registered [ 1230.850970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1238.839590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1238.874513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.505182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.693690] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1250.794957] LustreError: 42704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.813282] LustreError: 42704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1254.591538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1255.061931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1259.859961] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1266.783442] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1270.932297] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1277.962399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1278.387805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.850265] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1285.561071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1290.762627] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1298.635883] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1300.960307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1300.966643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.973053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.011081] Lustre: Skipped 2 previous similar messages [ 1303.717642] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1303.730586] Lustre: Skipped 2 previous similar messages [ 1303.804964] LustreError: 44947:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1304.035457] Lustre: server umount lustre-MDT0000 complete [ 1306.086627] LustreError: 42704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.117558] LustreError: 42704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1308.120051] LustreError: 44948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555272 with bad export cookie 5040022245817606257 [ 1308.132821] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.840925] LustreError: 44432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.898115] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 1311.200363] LustreError: 44432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.206984] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.248395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1313.258823] Lustre: Skipped 1 previous similar message [ 1314.379495] LustreError: 45149:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1314.391626] LustreError: 45149:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1314.867396] Lustre: server umount lustre-MDT0001 complete [ 1336.281321] LustreError: 45373:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1336.284562] LustreError: 45373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1336.469813] Lustre: server umount lustre-OST0000 complete [ 1355.596993] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1359.330253] Key type lgssc unregistered [ 1359.985741] LNet: 46211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.030843] LNetError: 46211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.064835] LNet: Removed LNI 192.168.202.133@tcp [ 1361.580066] Key type .llcrypt unregistered [ 1361.581741] Key type ._llcrypt unregistered [ 1392.309456] Key type ._llcrypt registered [ 1392.311123] Key type .llcrypt registered [ 1392.485076] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1393.877633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.919653] alg: No test for adler32 (adler32-zlib) [ 1394.962820] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1395.170985] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1396.839996] Key type lgssc registered [ 1398.124547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1406.531420] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1406.547758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1422.111772] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1427.231415] LustreError: 47581:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1453.920365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.308051] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1469.134279] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1635.566219] LustreError: 48289:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1635.912353] Lustre: server umount lustre-OST0000 complete [ 1655.954738] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1659.246163] Key type lgssc unregistered [ 1659.529642] LNet: 49127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.545972] LNetError: 49127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.594060] LNet: Removed LNI 192.168.202.133@tcp [ 1661.487480] Key type .llcrypt unregistered [ 1661.496382] Key type ._llcrypt unregistered [ 1686.153033] Key type ._llcrypt registered [ 1686.155498] Key type .llcrypt registered [ 1686.249942] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1687.655263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1687.748173] alg: No test for adler32 (adler32-zlib) [ 1688.800275] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1689.154556] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1690.927858] Key type lgssc registered [ 1692.561997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.683095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.705269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.592684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.167870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1715.233240] LustreError: 50521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.255282] LustreError: 50521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1720.298032] LustreError: 50521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.637963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1722.899117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1728.908096] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1737.230649] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1742.841918] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1750.888930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1751.139707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.275419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1763.387715] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1766.647271] LustreError: 52761:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1766.775526] Lustre: server umount lustre-OST0000 complete [ 1773.028412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.050266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.059216] Lustre: Skipped 1 previous similar message [ 1777.193480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.199316] Lustre: Skipped 1 previous similar message [ 1782.241107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.311162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1785.441393] LustreError: 52964:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1785.445734] LustreError: 52964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1785.664544] Lustre: server umount lustre-MDT0000 complete [ 1787.365217] LustreError: 50518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.380730] LustreError: 50518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1789.413721] LustreError: 50502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555753 with bad export cookie 1587717840786278351 [ 1789.431580] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.664347] LustreError: 53166:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1789.668583] LustreError: 53166:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1790.104783] Lustre: server umount lustre-MDT0001 complete [ 1800.360042] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1802.975988] Key type lgssc unregistered [ 1803.396411] LNet: 53698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.401990] LNetError: 53698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.427498] LNet: Removed LNI 192.168.202.133@tcp [ 1804.313218] Key type .llcrypt unregistered [ 1804.314574] Key type ._llcrypt unregistered [ 1830.466637] Key type ._llcrypt registered [ 1830.469354] Key type .llcrypt registered [ 1830.592515] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1831.923080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.159351] alg: No test for adler32 (adler32-zlib) [ 1833.288492] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1833.695896] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1835.447190] Key type lgssc registered [ 1836.915550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.301260] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.340034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1861.023775] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1866.143651] LustreError: 55078:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1892.831443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.446822] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1906.741312] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1914.464133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.940901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.040925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1918.496548] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1919.969245] LustreError: 55104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.992314] LustreError: 55104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1925.093242] LustreError: 55104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.016760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1926.308985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.296245] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1936.068881] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1939.865418] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1941.985075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1941.998916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.015229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.009813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.012565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.130455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.148283] Lustre: Skipped 2 previous similar messages [ 1953.257371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.268907] Lustre: Skipped 1 previous similar message [ 1956.319128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1956.365564] LustreError: 57351:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1956.464647] Lustre: server umount lustre-OST0000 complete [ 1978.851293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.858436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.862893] Lustre: Skipped 1 previous similar message [ 1978.879858] Lustre: Skipped 1 previous similar message [ 1984.256808] LustreError: 57731:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1984.269416] LustreError: 57731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1984.595803] Lustre: server umount lustre-MDT0000 complete [ 1988.073708] LustreError: 56004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.095804] LustreError: 56004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1988.411435] LustreError: 57479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768555952 with bad export cookie 11117666992999280152 [ 1988.422200] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.426498] LustreError: 57479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1994.647694] LustreError: 57932:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1994.655774] LustreError: 57932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1994.820173] Lustre: server umount lustre-MDT0001 complete [ 2007.483634] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2010.282481] Key type lgssc unregistered [ 2010.686180] LNet: 58465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.696581] LNetError: 58465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.712924] LNet: Removed LNI 192.168.202.133@tcp [ 2011.702627] Key type .llcrypt unregistered [ 2011.704484] Key type ._llcrypt unregistered [ 2035.854165] Key type ._llcrypt registered [ 2035.855776] Key type .llcrypt registered [ 2035.949450] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2037.074782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.230623] alg: No test for adler32 (adler32-zlib) [ 2038.328563] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2038.590457] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2040.279165] Key type lgssc registered [ 2041.385709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.037997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2050.050654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.544681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.940263] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2061.804378] LustreError: 59862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.815214] LustreError: 59862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2064.221336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2064.500095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.529801] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2074.027129] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.545358] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2084.634186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2084.971745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.621488] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2096.253767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2097.155298] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2122.215467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2122.223201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.234420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2122.244026] Lustre: Skipped 1 previous similar message [ 2122.257103] Lustre: Skipped 1 previous similar message [ 2123.732148] LustreError: 62131:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2123.885915] Lustre: server umount lustre-OST0000 complete [ 2130.912648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.930471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.952298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.449209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.459453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.481122] Lustre: Skipped 1 previous similar message [ 2134.606293] LustreError: 62332:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2134.621054] LustreError: 62332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2134.791301] Lustre: server umount lustre-MDT0000 complete [ 2137.582867] LustreError: 59858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.612804] LustreError: 59858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2138.827944] LustreError: 59842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556102 with bad export cookie 8812017983407217118 [ 2138.835833] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.841640] LustreError: 59842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2139.021024] LustreError: 62534:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2139.027366] LustreError: 62534:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2139.392777] Lustre: server umount lustre-MDT0001 complete [ 2149.980114] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2152.407394] Key type lgssc unregistered [ 2152.744202] LNet: 63068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.774214] LNetError: 63068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.799217] LNet: Removed LNI 192.168.202.133@tcp [ 2153.620308] Key type .llcrypt unregistered [ 2153.622869] Key type ._llcrypt unregistered [ 2180.111050] Key type ._llcrypt registered [ 2180.116869] Key type .llcrypt registered [ 2180.290878] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2181.714504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.794338] alg: No test for adler32 (adler32-zlib) [ 2182.956707] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2183.292134] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2185.015183] Key type lgssc registered [ 2186.286990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.998202] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:35:58 (1768556158) [ 2205.714596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2205.754145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.427690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.063599] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2217.965875] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.986124] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2223.072580] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.519684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2224.871889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.398391] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2235.923485] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2240.477406] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2248.609194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2249.043725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.189375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2256.018909] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2262.324763] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2269.328076] Lustre: Failing over lustre-MDT0000 [ 2269.549124] LustreError: 66996:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2269.714589] Lustre: server umount lustre-MDT0000 complete [ 2270.395535] LustreError: 64750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.201161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2271.224742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.772369] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.781172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.796212] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2273.799679] Lustre: Skipped 1 previous similar message [ 2278.880516] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.897660] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2279.361435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.561990] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.667471] Lustre: *** cfs_fail_loc=135, val=0*** [ 2279.768928] LustreError: 67505:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2279.776770] LustreError: 67505:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -2 [ 2279.797422] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2279.797422] [ 2279.814996] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2279.836708] LustreError: 67494:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2279.844677] LustreError: 67494:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 2279.850701] LustreError: 67494:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2279.857746] LustreError: 67494:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2279.895954] Lustre: server umount lustre-MDT0000 complete [ 2279.897975] LustreError: 67494:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2287.155698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.325149] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.340165] LustreError: 64747:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2287.347201] LustreError: 67850:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.202.133@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000001e679306) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2287.789848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.853447] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2291.873760] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2292.326911] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2293.245712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2293.257771] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2293.323675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2301.416083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2301.427517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.442439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2303.461968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2303.463026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.579521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.589252] Lustre: Skipped 1 previous similar message [ 2313.701517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.716403] Lustre: Skipped 1 previous similar message [ 2315.231325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2315.295327] LustreError: 68312:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2315.306033] LustreError: 68312:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2315.437914] Lustre: server umount lustre-OST0000 complete [ 2318.817082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.826557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2318.840984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.840992] Lustre: Skipped 1 previous similar message [ 2329.059265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.091033] Lustre: Skipped 4 previous similar messages [ 2333.152699] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2333.288481] LustreError: 68514:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2333.300741] LustreError: 68514:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2333.708802] Lustre: server umount lustre-MDT0000 complete [ 2334.180389] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.207835] LustreError: 64755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2337.656809] LustreError: 65472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556301 with bad export cookie 3587815839234487198 [ 2337.660327] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.668945] LustreError: 65472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2337.870380] LustreError: 68717:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2337.890833] LustreError: 68717:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2338.071858] Lustre: server umount lustre-MDT0001 complete [ 2349.987353] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2352.580905] Key type lgssc unregistered [ 2352.848146] LNet: 69250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.859982] LNetError: 69250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.880948] LNet: Removed LNI 192.168.202.133@tcp [ 2353.628659] Key type .llcrypt unregistered [ 2353.630592] Key type ._llcrypt unregistered [ 2378.821410] Key type ._llcrypt registered [ 2378.824267] Key type .llcrypt registered [ 2378.936782] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2380.315361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.419891] alg: No test for adler32 (adler32-zlib) [ 2381.528159] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2381.727664] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2383.407820] Key type lgssc registered [ 2384.604025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2392.993812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2393.017053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.759497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.673069] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2404.339881] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.369685] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2408.970763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2409.331826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2414.092457] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2420.059705] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2424.283868] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2432.036994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2432.268820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.232346] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2439.472234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2443.687611] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2450.968183] Lustre: Failing over lustre-MDT0000 [ 2451.129557] LustreError: 72887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2451.290713] Lustre: server umount lustre-MDT0000 complete [ 2451.650443] LustreError: 70639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.017721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.028751] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.041517] Lustre: Skipped 1 previous similar message [ 2455.080858] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2460.148784] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.182536] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2460.736691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.812625] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.928126] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2461.111695] LustreError: 73396:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2461.128649] LustreError: 73396:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -9 [ 2461.141593] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2461.141593] [ 2461.156982] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2461.164447] LustreError: 73385:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2461.175914] LustreError: 73385:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -9 [ 2461.180871] LustreError: 73385:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2461.187728] LustreError: 73385:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2461.201829] Lustre: server umount lustre-MDT0000 complete [ 2461.205186] LustreError: 73385:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2465.249896] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.286800] LustreError: 70643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2469.511239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.623474] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.836164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.885864] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2473.128314] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2475.011223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2475.027292] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2475.091635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2475.441120] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2485.732482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2485.754934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.769258] Lustre: Skipped 1 previous similar message [ 2485.775854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2485.785954] Lustre: Skipped 1 previous similar message [ 2488.304947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.315249] Lustre: Skipped 1 previous similar message [ 2490.821716] LustreError: 74206:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2490.826831] LustreError: 74206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2490.981560] Lustre: server umount lustre-OST0000 complete [ 2495.459588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2495.465890] LustreError: Skipped 1 previous similar message [ 2495.472853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.494037] Lustre: Skipped 1 previous similar message [ 2495.503024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.528213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.528534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.649160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.659417] Lustre: Skipped 2 previous similar messages [ 2509.280065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2509.373195] LustreError: 74408:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2509.387447] LustreError: 74408:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2509.469830] Lustre: server umount lustre-MDT0000 complete [ 2513.600557] LustreError: 70625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556477 with bad export cookie 8199516115537621383 [ 2513.608445] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.614647] LustreError: 70625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2513.767357] LustreError: 74610:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2513.775292] LustreError: 74610:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2513.943296] Lustre: server umount lustre-MDT0001 complete [ 2527.788286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2531.230445] Key type lgssc unregistered [ 2531.528966] LNet: 75142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.535024] LNetError: 75142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.587240] LNet: Removed LNI 192.168.202.133@tcp [ 2533.410624] Key type .llcrypt unregistered [ 2533.413261] Key type ._llcrypt unregistered [ 2547.117798] Key type ._llcrypt registered [ 2547.125287] Key type .llcrypt registered [ 2547.249709] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2547.491988] Key type .llcrypt unregistered [ 2547.497419] Key type ._llcrypt unregistered [ 2571.221253] Key type ._llcrypt registered [ 2571.222800] Key type .llcrypt registered [ 2571.307203] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2572.611298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2572.684034] alg: No test for adler32 (adler32-zlib) [ 2573.915819] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2574.225979] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2575.935185] Key type lgssc registered [ 2577.126851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2584.614563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2584.638673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.264604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.237451] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2596.840853] LustreError: 77005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.879495] LustreError: 77005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2599.272448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2599.596410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.001360] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2610.255412] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2614.723627] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2622.304464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2622.771174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.759611] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2630.967767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2636.452272] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2646.502386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.503819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.522731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.538356] Lustre: Skipped 1 previous similar message [ 2646.558221] Lustre: Skipped 1 previous similar message [ 2651.623676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.628211] Lustre: Skipped 1 previous similar message [ 2652.296224] LustreError: 79250:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2652.514952] Lustre: server umount lustre-OST0000 complete [ 2656.736598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.751867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.345087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2661.350747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.386913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.862768] LustreError: 79451:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2662.874308] LustreError: 79451:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2663.048864] Lustre: server umount lustre-MDT0000 complete [ 2666.980097] LustreError: 77000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.013464] LustreError: 77000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2667.082323] LustreError: 76987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556631 with bad export cookie 12798577504095858571 [ 2667.088960] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.092338] LustreError: 76987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2667.268559] LustreError: 79654:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2667.281450] LustreError: 79654:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2667.465251] Lustre: server umount lustre-MDT0001 complete [ 2679.573706] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2681.848455] Key type lgssc unregistered [ 2682.158554] LNet: 80187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.163517] LNetError: 80187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.177135] LNet: Removed LNI 192.168.202.133@tcp [ 2682.884441] Key type .llcrypt unregistered [ 2682.887485] Key type ._llcrypt unregistered [ 2706.885552] Key type ._llcrypt registered [ 2706.890350] Key type .llcrypt registered [ 2706.995900] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2708.466840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2708.485728] alg: No test for adler32 (adler32-zlib) [ 2709.601486] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2709.814532] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2711.495481] Key type lgssc registered [ 2712.849447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2721.047473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2721.074115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.480085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.165837] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2732.024183] LustreError: 81578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.053639] LustreError: 81578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2734.292115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2734.801404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2738.747587] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2744.169203] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2747.767542] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2754.213964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2754.601253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.861465] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2765.156385] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2767.925554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2778.088433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.095171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.104135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.116204] Lustre: Skipped 1 previous similar message [ 2778.128778] Lustre: Skipped 1 previous similar message [ 2783.219380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.231284] Lustre: Skipped 1 previous similar message [ 2783.996691] LustreError: 83839:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2784.091855] Lustre: server umount lustre-OST0000 complete [ 2788.321803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.345319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.355293] Lustre: Skipped 1 previous similar message [ 2793.447246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.455874] Lustre: Skipped 1 previous similar message [ 2794.513621] LustreError: 84043:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2794.519255] LustreError: 84043:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2794.614280] Lustre: server umount lustre-MDT0000 complete [ 2798.562027] LustreError: 81578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.583145] LustreError: 81578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2798.939550] LustreError: 83193:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556762 with bad export cookie 1493901330649150157 [ 2798.950045] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.959835] LustreError: 83193:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2799.131275] LustreError: 84244:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2799.139838] LustreError: 84244:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2799.356487] Lustre: server umount lustre-MDT0001 complete [ 2814.682765] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2818.421501] Key type lgssc unregistered [ 2818.732437] LNet: 84777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2818.738863] LNetError: 84777:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2818.753211] LNet: Removed LNI 192.168.202.133@tcp [ 2819.502356] Key type .llcrypt unregistered [ 2819.504513] Key type ._llcrypt unregistered [ 2845.600801] Key type ._llcrypt registered [ 2845.605468] Key type .llcrypt registered [ 2845.718884] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2846.576348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2846.656261] alg: No test for adler32 (adler32-zlib) [ 2847.726764] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2847.949386] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2849.623151] Key type lgssc registered [ 2850.540816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2858.021333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2858.055209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.679149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.609125] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2869.220504] LustreError: 86170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.243595] LustreError: 86170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2873.628966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2874.215056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.933887] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2884.735524] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.288251] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2896.133256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.677252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.046860] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2904.898654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2908.892315] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2925.536763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2925.537808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.547074] LustreError: Skipped 1 previous similar message [ 2925.554781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.564975] Lustre: Skipped 1 previous similar message [ 2925.587135] Lustre: Skipped 1 previous similar message [ 2928.700455] LustreError: 88444:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2928.931120] Lustre: server umount lustre-OST0000 complete [ 2935.781347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2935.784908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.803514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.842403] Lustre: Skipped 1 previous similar message [ 2939.236652] LustreError: 88646:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2939.239498] LustreError: 88646:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2939.484421] Lustre: server umount lustre-MDT0000 complete [ 2940.899984] LustreError: 86170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.923882] LustreError: 86170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2943.487369] LustreError: 86151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768556907 with bad export cookie 7817084089475296299 [ 2943.498803] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.505700] LustreError: 86151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2943.706707] LustreError: 88847:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2943.723966] LustreError: 88847:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2943.910497] Lustre: server umount lustre-MDT0001 complete [ 2954.742316] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2957.677366] Key type lgssc unregistered [ 2958.238501] LNet: 89379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2958.251462] LNetError: 89379:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2958.265711] LNet: Removed LNI 192.168.202.133@tcp [ 2959.269130] Key type .llcrypt unregistered [ 2959.271638] Key type ._llcrypt unregistered [ 2985.166625] Key type ._llcrypt registered [ 2985.168517] Key type .llcrypt registered [ 2985.302859] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2986.913942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2986.963707] alg: No test for adler32 (adler32-zlib) [ 2988.157565] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2988.407420] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2990.088059] Key type lgssc registered [ 2991.539755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2999.887312] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2999.914577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3015.519915] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3020.639237] LustreError: 90748:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3047.393177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.927274] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3057.531416] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3062.748770] LustreError: 91603:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3062.831457] Lustre: server umount lustre-OST0000 complete [ 3070.593319] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:50:33 (1768557033) [ 3076.608079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.901070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.559168] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3082.217857] LustreError: 92272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.227653] LustreError: 92272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3086.310142] LustreError: 92272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.264463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3088.651457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3092.297778] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3097.640735] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.885856] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3106.490901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3106.751298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.286307] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3117.658093] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3121.078326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3136.493104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.494930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3136.523030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3136.540499] Lustre: Skipped 1 previous similar message [ 3140.833330] LustreError: 94675:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3140.837956] LustreError: 94675:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3140.955189] Lustre: server umount lustre-OST0000 complete [ 3145.191905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3145.214684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.223202] Lustre: Skipped 1 previous similar message [ 3145.226773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3146.720576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.721847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3146.735129] Lustre: Skipped 1 previous similar message [ 3150.334973] LustreError: 94876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3150.339372] LustreError: 94876:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3150.448679] Lustre: server umount lustre-MDT0000 complete [ 3151.841970] LustreError: 92985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.852410] LustreError: 92985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3153.965888] LustreError: 92254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557118 with bad export cookie 8711346809263077955 [ 3153.973607] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3153.976392] LustreError: 92254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3154.206673] LustreError: 95078:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3154.212533] LustreError: 95078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3154.426541] Lustre: server umount lustre-MDT0001 complete [ 3163.701992] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3166.342344] Key type lgssc unregistered [ 3166.616994] LNet: 95610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3166.621515] LNetError: 95610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3166.632970] LNet: Removed LNI 192.168.202.133@tcp [ 3167.467199] Key type .llcrypt unregistered [ 3167.471476] Key type ._llcrypt unregistered [ 3194.108166] Key type ._llcrypt registered [ 3194.115586] Key type .llcrypt registered [ 3194.285864] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3195.974367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.109391] alg: No test for adler32 (adler32-zlib) [ 3197.184547] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3197.412787] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3199.087367] Key type lgssc registered [ 3200.108766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3207.556185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3207.574944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.005913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.063862] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3219.427460] LustreError: 97006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.462294] LustreError: 97006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3221.182219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3221.581138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.937258] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3232.001543] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.748935] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.017705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3242.408864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.995664] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3253.077461] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3254.713362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3270.112194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3270.115860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.119932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3270.146148] Lustre: Skipped 1 previous similar message [ 3273.596147] LustreError: 99278:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3273.835070] Lustre: server umount lustre-OST0000 complete [ 3280.352838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.353898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.387218] Lustre: Skipped 1 previous similar message [ 3280.389868] Lustre: Skipped 1 previous similar message [ 3284.324618] LustreError: 99479:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3284.342215] LustreError: 99479:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3284.406736] Lustre: server umount lustre-MDT0000 complete [ 3285.472435] LustreError: 97717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.967987] LustreError: 96986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557252 with bad export cookie 7512597070701858579 [ 3287.986494] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.139943] LustreError: 99680:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3288.143959] LustreError: 99680:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3288.371765] Lustre: server umount lustre-MDT0001 complete [ 3298.673945] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3301.565883] Key type lgssc unregistered [ 3301.800628] LNet: 100212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.804954] LNetError: 100212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.816044] LNet: Removed LNI 192.168.202.133@tcp [ 3302.520698] Key type .llcrypt unregistered [ 3302.522789] Key type ._llcrypt unregistered [ 3322.405537] Key type ._llcrypt registered [ 3322.408914] Key type .llcrypt registered [ 3322.489659] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3323.204590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3323.274343] alg: No test for adler32 (adler32-zlib) [ 3324.463734] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3325.049473] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3326.881954] Key type lgssc registered [ 3328.000536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3337.745579] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3337.768558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3353.311355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3358.433679] LustreError: 101438:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3385.183177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.377782] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3397.475938] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3404.542689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.663076] LustreError: 100774:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000077db6eeb x1854466727872768/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3404.818227] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3404.828625] LustreError: 102349:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3404.836709] LustreError: 102349:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 3404.844834] LustreError: 102349:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3404.941810] Lustre: server umount lustre-MDT0000 complete [ 3404.944546] LustreError: 102349:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3425.247277] Lustre: 100778:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768557373/real 1768557373] req@0000000045cdd2c0 x1854466727873792/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768557389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3425.278913] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.287643] LustreError: 101450:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3425.330547] LustreError: 102778:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3425.453592] Lustre: server umount lustre-OST0000 complete [ 3434.760880] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3445.377542] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3486.676283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.995138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3487.025860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3487.071588] Lustre: lustre-MDT0000: new disk, initializing [ 3487.135276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.145705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3492.419786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3504.851154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3504.995768] Lustre: 105106:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3505.093167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3505.095640] Lustre: Skipped 1 previous similar message [ 3505.220647] Lustre: lustre-MDT0001: new disk, initializing [ 3505.379302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3505.426801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3505.443626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3510.914257] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3518.512656] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.489883] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3529.666125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3529.887481] Lustre: lustre-OST0000: new disk, initializing [ 3529.895900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3530.001904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.451105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3535.922502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3535.950111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3536.023547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3543.243835] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3549.167852] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3549.386934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3554.773931] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3555.026645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3561.957523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.962219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.996554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3563.331336] LustreError: 108039:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3563.343390] LustreError: 108039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3563.502733] Lustre: server umount lustre-OST0000 complete [ 3572.192692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3572.197811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3572.213939] Lustre: Skipped 1 previous similar message [ 3572.219364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3572.253744] Lustre: Skipped 2 previous similar messages [ 3574.144825] LustreError: 108240:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3574.159926] LustreError: 108240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3574.319824] Lustre: server umount lustre-MDT0000 complete [ 3578.863777] LustreError: 105096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557542 with bad export cookie 17058636059561825163 [ 3578.869128] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.888014] LustreError: 105096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3579.027910] LustreError: 108443:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3579.032313] LustreError: 108443:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3579.279939] Lustre: server umount lustre-MDT0001 complete [ 3588.788519] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:59:10 (1768557550) [ 3592.244328] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3609.521431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3621.946625] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3664.515775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.138634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3665.248623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3665.379215] Lustre: lustre-MDT0000: new disk, initializing [ 3665.477557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.500601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3672.064688] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3683.745573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3683.922825] Lustre: 111570:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3683.979543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3683.983692] Lustre: Skipped 1 previous similar message [ 3684.053200] Lustre: lustre-MDT0001: new disk, initializing [ 3684.124801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3684.157490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3684.173231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3689.156670] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3697.754323] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3702.954067] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3711.630579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3712.084937] Lustre: lustre-OST0000: new disk, initializing [ 3712.091243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3712.239831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.702365] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3719.302264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3719.313098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3719.502045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3727.896917] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3733.649844] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3733.826865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3739.446114] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3739.649520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3745.275769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.290856] Lustre: Skipped 1 previous similar message [ 3745.306444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3745.321110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3745.335565] Lustre: Skipped 1 previous similar message [ 3747.768977] LustreError: 114510:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3747.773269] LustreError: 114510:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3747.881815] Lustre: server umount lustre-OST0000 complete [ 3755.501883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.509210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3755.523475] Lustre: Skipped 2 previous similar messages [ 3758.341831] LustreError: 114712:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3758.353390] LustreError: 114712:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3758.701827] Lustre: server umount lustre-MDT0000 complete [ 3760.609576] LustreError: 113326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.626343] LustreError: 113326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3762.863456] LustreError: 111561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557726 with bad export cookie 17058636059561826045 [ 3762.865113] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3762.875413] LustreError: 111561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3763.321808] Lustre: server umount lustre-MDT0001 complete [ 3772.582861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.927266] LustreError: 115391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.984753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.783375] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3778.025501] LustreError: 115391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.143043] LustreError: 115391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.603993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3792.416332] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3799.273742] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3804.378504] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.769438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3819.899174] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3825.668599] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3833.469989] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3839.968469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3839.975073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.999060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.004985] Lustre: Skipped 1 previous similar message [ 3844.087599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.201367] LustreError: 117730:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3845.211340] LustreError: 117730:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3845.381906] Lustre: server umount lustre-OST0000 complete [ 3849.199184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.206805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3849.222830] Lustre: Skipped 2 previous similar messages [ 3849.243733] Lustre: Skipped 3 previous similar messages [ 3855.707872] Lustre: server umount lustre-MDT0000 complete [ 3859.431923] LustreError: 116604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3859.452624] LustreError: 116604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3859.786793] LustreError: 117008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768557823 with bad export cookie 17058636059561826920 [ 3859.793284] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3859.820912] LustreError: 117008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3860.296916] Lustre: server umount lustre-MDT0001 complete [ 3872.183501] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3874.802898] Key type lgssc unregistered [ 3875.075901] LNet: 118664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3875.088775] LNetError: 118664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3876.147292] LNet: Removed LNI 192.168.202.133@tcp [ 3876.925594] Key type .llcrypt unregistered [ 3876.929607] Key type ._llcrypt unregistered [ 3894.404584] Key type ._llcrypt registered [ 3894.407403] Key type .llcrypt registered [ 3894.527962] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3910.272706] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3911.043616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3911.059919] alg: No test for adler32 (adler32-zlib) [ 3912.175739] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 3912.384961] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3914.016546] Key type lgssc registered [ 3914.944054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3948.088529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3948.144248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.532682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3949.594910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3949.761255] Lustre: lustre-MDT0000: new disk, initializing [ 3949.857899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.868584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3954.197623] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3965.184232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3965.299122] Lustre: 121783:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3965.416845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3965.423453] Lustre: Skipped 1 previous similar message [ 3965.557972] Lustre: lustre-MDT0001: new disk, initializing [ 3965.704928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.778334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3965.784538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3970.912644] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3980.336536] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3984.319398] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3991.242399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3991.547724] Lustre: lustre-OST0000: new disk, initializing [ 3991.551367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3991.636707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.789305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3997.798142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3997.953315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3998.226341] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4005.839298] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4012.082639] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4012.381742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4017.129579] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4017.394271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4023.779438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4023.781931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.786953] LustreError: Skipped 1 previous similar message [ 4023.791795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4023.798037] Lustre: Skipped 1 previous similar message [ 4023.828915] Lustre: Skipped 1 previous similar message [ 4025.725301] LustreError: 124716:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4026.013350] Lustre: server umount lustre-OST0000 complete [ 4032.480773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4032.486267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.497850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4034.018497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4034.018683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.782667] LustreError: 124918:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4036.790662] LustreError: 124918:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4036.977647] Lustre: server umount lustre-MDT0000 complete [ 4040.363601] LustreError: 121777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558004 with bad export cookie 4073660302520047167 [ 4040.364390] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.386933] LustreError: 121777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4040.569298] LustreError: 125119:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4040.589159] LustreError: 125119:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4040.777344] Lustre: server umount lustre-MDT0001 complete [ 4050.750524] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:06:53 (1768558013) [ 4057.854907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.415843] LustreError: 125790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.435810] LustreError: 125790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4058.533864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.077842] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4063.724474] LustreError: 125790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.845734] LustreError: 125790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.536411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4072.980796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.603461] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4084.677598] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4090.233328] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4093.408297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.427699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.430554] Lustre: Skipped 1 previous similar message [ 4097.572666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.583282] Lustre: Skipped 1 previous similar message [ 4098.618869] LustreError: 127132:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4098.621586] LustreError: 127132:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4098.699514] Lustre: server umount lustre-MDT0000 complete [ 4102.629642] LustreError: 126501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4102.657365] LustreError: 126501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4102.907040] LustreError: 125770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558066 with bad export cookie 4073660302520048049 [ 4102.912787] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.922985] LustreError: 125770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4103.199120] LustreError: 127333:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4103.209133] LustreError: 127333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4103.695803] Lustre: server umount lustre-MDT0001 complete [ 4112.838282] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:07:55 (1768558075) [ 4119.218893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4134.754824] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4139.871912] LustreError: 127980:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4165.599382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.449689] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4176.634791] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4178.497716] LustreError: 128643:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4178.506514] LustreError: 128643:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4178.660776] Lustre: server umount lustre-OST0000 complete [ 4186.634897] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:09:09 (1768558149) [ 4193.381832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.907852] LustreError: 129315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.003745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.599487] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4203.488927] LustreError: 129315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.507463] LustreError: 129315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4208.572564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4214.234078] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4222.376771] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4226.683929] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4233.996694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4234.511292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.526722] Lustre: Skipped 1 previous similar message [ 4241.399896] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4247.290550] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4259.301944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.318787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.325485] Lustre: Skipped 3 previous similar messages [ 4259.343872] Lustre: Skipped 2 previous similar messages [ 4263.677870] LustreError: 131556:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4263.685155] LustreError: 131556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4263.809541] Lustre: server umount lustre-MDT0000 complete [ 4264.421379] LustreError: 129314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.453408] LustreError: 129314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4268.085055] LustreError: 129295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558232 with bad export cookie 4073660302520048714 [ 4268.088895] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.103286] LustreError: 129295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4269.064918] Lustre: server umount lustre-MDT0001 complete [ 4283.501573] Lustre: server umount lustre-OST0000 complete [ 4292.982868] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:10:55 (1768558255) [ 4298.756530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.090037] LustreError: 132634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.144171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.106087] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4312.117649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4316.035753] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4321.006763] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.374375] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4330.989592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4338.091394] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4344.417390] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4344.791276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4348.616748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4348.889385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4353.474249] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4353.624837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4358.118562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4358.132057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.144958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4369.375154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4369.424545] LustreError: 135374:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4369.427106] LustreError: 135374:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4369.715120] Lustre: server umount lustre-OST0000 complete [ 4373.984507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4375.525436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.545437] Lustre: Skipped 2 previous similar messages [ 4388.319559] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4388.632774] Lustre: server umount lustre-MDT0000 complete [ 4390.880756] LustreError: 132634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.892912] LustreError: 132634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4391.810430] LustreError: 132613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558355 with bad export cookie 4073660302520049785 [ 4391.822638] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.045872] Lustre: server umount lustre-MDT0001 complete [ 4398.826540] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:12:41 (1768558361) [ 4405.669074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4421.282664] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4426.399486] LustreError: 136423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4452.127623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.141970] Lustre: Skipped 2 previous similar messages [ 4457.517935] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4463.272762] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4470.276472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.876525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4475.173707] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4482.958681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4487.263990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4493.289949] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4497.759974] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4503.523690] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4503.903662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4509.436315] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4509.636934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4512.229957] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.239403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.246975] Lustre: Skipped 1 previous similar message [ 4512.271554] Lustre: Skipped 13 previous similar messages [ 4517.796978] LustreError: 139196:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4517.802214] LustreError: 139196:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4518.069749] Lustre: server umount lustre-MDT0000 complete [ 4522.075706] LustreError: 136431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558486 with bad export cookie 4073660302520050807 [ 4522.079331] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.102423] LustreError: 136431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4522.470273] LustreError: 137342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.492542] LustreError: 137342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4522.627761] Lustre: server umount lustre-MDT0001 complete [ 4528.848404] Lustre: server umount lustre-OST0000 complete [ 4536.496875] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:14:59 (1768558499) [ 4542.860531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4558.367613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4563.488492] LustreError: 140243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4596.328924] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4602.398388] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.098704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.991752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4617.546995] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4629.705514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4634.260506] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4642.247850] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4647.294215] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4654.518177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4654.675587] Lustre: lustre-OST0001: new disk, initializing [ 4654.682633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4661.156800] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4661.305776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4661.317650] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4661.440087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4670.272181] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4676.088797] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4676.371337] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4682.526316] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4682.939809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4686.327096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.337320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.347096] Lustre: Skipped 2 previous similar messages [ 4686.352785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.370861] Lustre: Skipped 3 previous similar messages [ 4700.127256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4700.254211] LustreError: 143971:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4700.273050] LustreError: 143971:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4700.359154] Lustre: server umount lustre-OST0000 complete [ 4707.809957] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4707.821330] LustreError: Skipped 1 previous similar message [ 4711.233953] Lustre: server umount lustre-OST0001 complete [ 4721.700012] Lustre: server umount lustre-MDT0000 complete [ 4725.163128] LustreError: 140250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558689 with bad export cookie 4073660302520051752 [ 4725.166910] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4725.172613] LustreError: 140250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4725.638730] Lustre: server umount lustre-MDT0001 complete [ 4750.097538] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:18:32 (1768558712) [ 4751.747897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4753.289882] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4755.287173] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:18:37 (1768558717) [ 4763.646098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.325291] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4764.375915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.693371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.706096] Lustre: Skipped 6 previous similar messages [ 4770.353622] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4780.285256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4780.419696] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4780.464509] Lustre: 146192:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4786.121520] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4792.761281] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4796.924756] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4803.683712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4803.871993] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4809.341470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4811.034850] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4816.296635] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4821.257861] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4821.505340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4826.011780] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4826.217808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4829.667670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.683690] LustreError: Skipped 1 previous similar message [ 4829.690109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.705205] Lustre: Skipped 5 previous similar messages [ 4829.713685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4829.722431] Lustre: Skipped 12 previous similar messages [ 4834.526452] Lustre: server umount lustre-OST0000 complete [ 4860.265370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4867.315603] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4870.074551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.112781] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4872.641527] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4878.007904] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4878.176851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4880.358326] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.371197] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4882.476467] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4882.721424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4887.069517] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a5218366800.ost_server_uuid 50 [ 4888.639380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a5218366800.ost_server_uuid in FULL state after 0 sec [ 4888.865530] LustreError: 147837:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.33@tcp) returned error from blocking AST (req@00000000729cd162 x1854467343646976 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b625d9c3/0x38888c764351717b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.33@tcp remote: 0xb53987a2ff137e31 expref: 5 pid: 146201 timeout: 4988 lvb_type: 0 [ 4888.883743] LustreError: lustre-MDT0000: A client on nid 192.168.202.33@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4888.891803] LustreError: 146189:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.33@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b625d9c3/0x38888c764351717b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.33@tcp remote: 0xb53987a2ff137e31 expref: 6 pid: 146201 timeout: 0 lvb_type: 0 [ 4895.219054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4895.229151] LustreError: Skipped 1 previous similar message [ 4899.516915] Lustre: server umount lustre-OST0000 complete [ 4917.215530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4917.415293] Lustre: server umount lustre-MDT0000 complete [ 4920.288438] LustreError: 147166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.312161] LustreError: 147166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4921.305951] LustreError: 146933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768558885 with bad export cookie 4073660302520052928 [ 4921.313489] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.316992] LustreError: 146933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4927.837603] Lustre: server umount lustre-MDT0001 complete [ 4938.730724] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4941.801430] Key type lgssc unregistered [ 4942.177899] LNet: 151677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4942.192994] LNetError: 151677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4942.218553] LNet: Removed LNI 192.168.202.133@tcp [ 4943.213402] Key type .llcrypt unregistered [ 4943.216114] Key type ._llcrypt unregistered [ 4968.133073] Key type ._llcrypt registered [ 4968.134368] Key type .llcrypt registered [ 4968.215623] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4969.502307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4969.557735] alg: No test for adler32 (adler32-zlib) [ 4970.707689] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 4970.932950] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4972.623242] Key type lgssc registered [ 4973.963383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4981.527452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4981.544293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4983.261910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.023532] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4993.516748] LustreError: 153072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.535536] LustreError: 153072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4997.899408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4998.440446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.072945] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5010.212410] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5014.755546] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5022.314313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5022.588455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.407887] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5028.730497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5034.347742] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5040.692658] Lustre: Failing over lustre-MDT0000 [ 5040.890274] LustreError: 155314:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5041.041626] Lustre: server umount lustre-MDT0000 complete [ 5042.372172] LustreError: 153068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.196781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.205645] LustreError: 154685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.209457] Lustre: Skipped 1 previous similar message [ 5044.226665] LustreError: 154685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5049.315754] LustreError: 153069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.350646] LustreError: 153069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5052.982399] LustreError: 153054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.33@tcp arrived at 1768559017 with bad export cookie 13987638405014293628 [ 5054.448375] LustreError: 153072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.476756] LustreError: 153072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5060.396062] Lustre: 152385:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768559008/real 1768559008] req@00000000c11787c8 x1854468452803584/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768559024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5060.416254] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.869718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.709973] LustreError: 153072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.741942] LustreError: 153072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5070.818933] LustreError: 152381:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000067ec695e x1854468452808576/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5071.276812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.370968] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5076.467179] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5076.482310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5076.587373] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5079.425141] Lustre: lustre-MDT0000: Denying connection for new client 244b1236-31e6-4762-9166-17d9e5591e31 (at 192.168.202.33@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5090.272747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.285609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.312812] Lustre: Skipped 1 previous similar message [ 5090.335504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.810498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.821381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.930157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5096.933400] Lustre: Skipped 2 previous similar messages [ 5102.047821] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5102.052758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5102.093956] Lustre: Skipped 1 previous similar message [ 5102.269145] LustreError: 156247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5102.276148] LustreError: 156247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5102.457698] Lustre: server umount lustre-OST0000 complete [ 5106.855641] LustreError: 156450:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5106.862436] Lustre: 155814:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5106.869389] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5106.887271] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5106.907295] LustreError: 155814:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fd8c36aa x1854468452825600/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5106.926580] LustreError: 155814:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5106.935358] LustreError: 155814:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5106.943942] Lustre: lustre-MDT0000: Recovery over after 0:30, of 2 clients 0 recovered and 2 were evicted. [ 5107.167129] LustreError: 156450:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5107.179503] LustreError: 156450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5107.182148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.221649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5107.575318] Lustre: server umount lustre-MDT0000 complete [ 5111.638331] LustreError: 156248:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559075 with bad export cookie 13987638405014293726 [ 5111.644992] LustreError: 152381:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5111.645815] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.657620] LustreError: 156248:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5113.873965] LustreError: 156652:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5113.883361] LustreError: 156652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5114.190445] Lustre: server umount lustre-MDT0001 complete [ 5126.934029] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5130.106272] Key type lgssc unregistered [ 5130.542710] LNet: 157184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5130.549406] LNetError: 157184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5130.563916] LNet: Removed LNI 192.168.202.133@tcp [ 5131.540120] Key type .llcrypt unregistered [ 5131.542053] Key type ._llcrypt unregistered [ 5157.685494] Key type ._llcrypt registered [ 5157.695197] Key type .llcrypt registered [ 5157.888603] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5159.203504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5159.276268] alg: No test for adler32 (adler32-zlib) [ 5160.317504] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5160.515522] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5162.279198] Key type lgssc registered [ 5163.531944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5171.397751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5171.418505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5173.075573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5177.639230] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5183.462031] LustreError: 158578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.480024] LustreError: 158578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5186.547274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5186.916179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5190.976953] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5197.663727] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5201.516359] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5209.544718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5210.146694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.411888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5220.167853] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5227.072654] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5233.634236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.636033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5233.659718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.682646] Lustre: Skipped 1 previous similar message [ 5238.348788] LustreError: 160824:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5238.590821] Lustre: server umount lustre-OST0000 complete [ 5243.363834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5243.379821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.408523] Lustre: Skipped 1 previous similar message [ 5243.420365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5248.993109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5249.016218] Lustre: Skipped 2 previous similar messages [ 5253.088772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5253.092462] Lustre: Skipped 1 previous similar message [ 5257.183135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5257.280904] LustreError: 161026:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5257.292249] LustreError: 161026:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5257.417828] Lustre: server umount lustre-MDT0000 complete [ 5258.208636] LustreError: 160644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.231364] LustreError: 160644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5261.381938] LustreError: 158560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559225 with bad export cookie 2017569051101413802 [ 5261.387174] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.391413] LustreError: 158560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5261.555658] LustreError: 161229:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5261.561602] LustreError: 161229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5261.776140] Lustre: server umount lustre-MDT0001 complete [ 5273.472555] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5277.080583] Key type lgssc unregistered [ 5277.406755] LNet: 161763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5277.417648] LNetError: 161763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5277.436625] LNet: Removed LNI 192.168.202.133@tcp [ 5278.774076] Key type .llcrypt unregistered [ 5278.776406] Key type ._llcrypt unregistered [ 5315.731927] Key type ._llcrypt registered [ 5315.734141] Key type .llcrypt registered [ 5315.850600] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5317.556493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5317.647513] alg: No test for adler32 (adler32-zlib) [ 5318.828512] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5319.187048] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5320.879340] Key type lgssc registered [ 5322.615666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.849597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5331.885420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.547353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.101597] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5343.713475] LustreError: 163450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.736502] LustreError: 163450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5347.319632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5347.625118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5352.233295] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5357.872942] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5362.229254] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5370.301482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5370.781637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5376.784257] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5377.987982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5383.572701] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5393.377123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5393.379573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.397402] LustreError: Skipped 1 previous similar message [ 5393.400081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5393.414986] Lustre: Skipped 1 previous similar message [ 5396.478418] LustreError: 165693:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5396.578530] Lustre: server umount lustre-OST0000 complete [ 5403.616898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.619475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5403.637487] Lustre: Skipped 1 previous similar message [ 5403.652272] Lustre: Skipped 2 previous similar messages [ 5406.825386] LustreError: 165894:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5406.832899] LustreError: 165894:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5406.958603] Lustre: server umount lustre-MDT0000 complete [ 5408.744925] LustreError: 165605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.775623] LustreError: 165605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5410.379838] LustreError: 163431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559374 with bad export cookie 13634477012452368961 [ 5410.381148] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5410.388210] LustreError: 163431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5410.525255] LustreError: 166096:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5410.531267] LustreError: 166096:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5410.661292] Lustre: server umount lustre-MDT0001 complete [ 5423.788543] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5426.670834] Key type lgssc unregistered [ 5426.997475] LNet: 166630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5427.004035] LNetError: 166630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5427.026395] LNet: Removed LNI 192.168.202.133@tcp [ 5428.034109] Key type .llcrypt unregistered [ 5428.035737] Key type ._llcrypt unregistered [ 5455.290375] Key type ._llcrypt registered [ 5455.292415] Key type .llcrypt registered [ 5455.392431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5456.486284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5456.530367] alg: No test for adler32 (adler32-zlib) [ 5457.736594] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5457.938274] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5459.584235] Key type lgssc registered [ 5460.818991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5471.809251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5471.826293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.429087] Lustre: *** cfs_fail_loc=135, val=0*** [ 5473.577032] LustreError: 168137:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 5473.589929] LustreError: 168137:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -2 [ 5473.597688] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5473.597688] [ 5473.628201] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5473.638293] LustreError: 168098:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5473.648582] LustreError: 168098:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 5473.662063] LustreError: 168098:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 5473.667894] LustreError: 168098:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 5473.680678] LustreError: 168098:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5473.762178] Lustre: server umount lustre-MDT0000 complete [ 5473.769672] LustreError: 168098:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5481.601371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.216398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.975406] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5492.706260] LustreError: 168508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.719223] LustreError: 168508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5495.740762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5496.059359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5500.316472] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5506.638219] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5510.752550] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5516.769133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5516.774280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.787801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.792576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.794130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.020599] LustreError: 169850:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5519.023991] LustreError: 169850:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5519.216859] Lustre: server umount lustre-MDT0000 complete [ 5521.953312] LustreError: 169496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.983117] LustreError: 169496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5522.738676] LustreError: 168490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559486 with bad export cookie 16946440852130889668 [ 5522.743076] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5522.752079] LustreError: 168490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5522.914911] LustreError: 170053:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5522.922149] LustreError: 170053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5523.272545] Lustre: server umount lustre-MDT0001 complete [ 5533.038187] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5535.197191] Key type lgssc unregistered [ 5535.444625] LNet: 170586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5535.448453] LNetError: 170586:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5535.468798] LNet: Removed LNI 192.168.202.133@tcp [ 5536.198891] Key type .llcrypt unregistered [ 5536.202261] Key type ._llcrypt unregistered [ 5558.641120] Key type ._llcrypt registered [ 5558.647855] Key type .llcrypt registered [ 5558.753228] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5559.130315] Key type .llcrypt unregistered [ 5559.133224] Key type ._llcrypt unregistered [ 5575.458679] Key type ._llcrypt registered [ 5575.462396] Key type .llcrypt registered [ 5575.556335] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5576.559768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5576.653328] alg: No test for adler32 (adler32-zlib) [ 5577.670501] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5577.877946] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5579.551272] Key type lgssc registered [ 5580.618780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5588.469652] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5588.501503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5604.064873] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5609.250991] LustreError: 172672:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 5636.063201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.613092] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5648.564736] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5654.817100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.935941] LustreError: 172008:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000374b89dc x1854469089265920/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5655.481090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.580190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5659.465632] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5660.646483] LustreError: 172697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.663625] LustreError: 172697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5665.768648] LustreError: 172697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5669.273635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5669.690682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.944629] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5680.898152] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5684.720641] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5687.981362] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5700.576585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5700.585289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.599970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.603891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.621281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.633476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.659291] Lustre: Skipped 1 previous similar message [ 5703.618144] LustreError: 175294:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5703.739419] Lustre: server umount lustre-OST0000 complete [ 5710.818959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5710.827646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.854594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5714.023608] LustreError: 175497:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5714.034467] LustreError: 175497:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5714.206317] Lustre: server umount lustre-MDT0000 complete [ 5716.961220] LustreError: 173596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5716.985795] LustreError: 173596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5717.935320] LustreError: 172679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559681 with bad export cookie 3047174656060581267 [ 5717.938508] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5717.951168] LustreError: 172679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5718.095883] LustreError: 175698:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5718.105296] LustreError: 175698:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5718.451474] Lustre: server umount lustre-MDT0001 complete [ 5730.795913] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5734.145875] Key type lgssc unregistered [ 5734.498569] LNet: 176231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5734.509345] LNetError: 176231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5734.522781] LNet: Removed LNI 192.168.202.133@tcp [ 5735.418275] Key type .llcrypt unregistered [ 5735.423648] Key type ._llcrypt unregistered [ 5760.745811] Key type ._llcrypt registered [ 5760.752628] Key type .llcrypt registered [ 5760.933601] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5762.272784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5762.285642] alg: No test for adler32 (adler32-zlib) [ 5763.599973] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5763.891211] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5765.551174] Key type lgssc registered [ 5767.158982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5775.823918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5775.857872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.660262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.532586] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5787.116091] LustreError: 177623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.144525] LustreError: 177623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5791.804564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5792.308615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5796.951555] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5803.783757] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5807.511304] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5815.046234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5815.454719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.677297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5822.757450] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5828.932303] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5835.665335] Lustre: Failing over lustre-MDT0000 [ 5835.830508] LustreError: 179921:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5835.976958] Lustre: server umount lustre-MDT0000 complete [ 5837.013095] LustreError: 177619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.305034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.309247] LustreError: 177623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.336602] Lustre: Skipped 2 previous similar messages [ 5838.363797] LustreError: 177623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5842.086738] LustreError: 177618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.107320] LustreError: 177618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5847.206563] LustreError: 178736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.222111] LustreError: 178736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5854.770290] Lustre: 176936:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768559802/real 1768559802] req@000000003fc66389 x1854469284323328/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1768559818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5854.803524] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.858523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.447216] LustreError: 177618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.481486] LustreError: 177618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5864.827854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.918299] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5867.682629] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5869.232249] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5870.066237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5870.110217] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5870.170753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5874.600803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5882.673475] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5894.117910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5894.124178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.139963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.633664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.646784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.651163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.658944] Lustre: Skipped 1 previous similar message [ 5897.951727] LustreError: 181211:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5897.965256] LustreError: 181211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5898.211942] Lustre: server umount lustre-OST0000 complete [ 5905.889236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.895244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5905.909357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.914845] Lustre: Skipped 2 previous similar messages [ 5908.246334] LustreError: 181412:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5908.252955] LustreError: 181412:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5908.420238] Lustre: server umount lustre-MDT0000 complete [ 5911.008717] LustreError: 177623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.027843] LustreError: 177623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5912.946451] LustreError: 181104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768559877 with bad export cookie 10381901040790183789 [ 5912.953111] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5912.958067] LustreError: 181104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5913.121981] LustreError: 181614:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5913.131710] LustreError: 181614:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5913.276575] Lustre: server umount lustre-MDT0001 complete [ 5925.990327] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5929.305332] Key type lgssc unregistered [ 5929.698443] LNet: 182146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5929.713441] LNetError: 182146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5929.752191] LNet: Removed LNI 192.168.202.133@tcp [ 5930.879180] Key type .llcrypt unregistered [ 5930.882893] Key type ._llcrypt unregistered [ 5958.900880] Key type ._llcrypt registered [ 5958.902847] Key type .llcrypt registered [ 5959.033717] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5960.039425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.119934] alg: No test for adler32 (adler32-zlib) [ 5961.521874] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 5961.771521] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5963.544089] Key type lgssc registered [ 5965.050390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5973.719495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5973.766176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5975.493228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.768319] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5985.771815] LustreError: 183538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.798829] LustreError: 183538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5990.885203] LustreError: 183538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.944730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5991.451447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5996.799987] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6002.819535] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6006.978286] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6013.842155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6014.176780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.771237] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6020.367802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6025.917745] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6030.002573] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6034.365367] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6043.076070] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6061.537676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.539247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6061.561315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.577783] Lustre: Skipped 1 previous similar message [ 6063.632783] LustreError: 185969:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6064.142973] Lustre: server umount lustre-OST0000 complete [ 6071.785966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.792076] Lustre: Skipped 2 previous similar messages [ 6071.792395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.825332] Lustre: Skipped 1 previous similar message [ 6076.903734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.993390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.996400] Lustre: Skipped 2 previous similar messages [ 6083.039147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6083.140615] LustreError: 186170:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6083.152197] LustreError: 186170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6083.304030] Lustre: server umount lustre-MDT0000 complete [ 6086.117892] LustreError: 183539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.136121] LustreError: 183539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6087.282460] LustreError: 183519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560051 with bad export cookie 17087384297913131559 [ 6087.290258] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.298084] LustreError: 183519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6087.426211] LustreError: 186373:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6087.431040] LustreError: 186373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6087.564898] Lustre: server umount lustre-MDT0001 complete [ 6098.157148] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6100.979626] Key type lgssc unregistered [ 6101.419312] LNet: 186906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6101.428332] LNetError: 186906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6101.462722] LNet: Removed LNI 192.168.202.133@tcp [ 6102.204887] Key type .llcrypt unregistered [ 6102.206795] Key type ._llcrypt unregistered [ 6127.935611] Key type ._llcrypt registered [ 6127.937499] Key type .llcrypt registered [ 6128.034885] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6129.340660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6129.403463] alg: No test for adler32 (adler32-zlib) [ 6130.402370] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 6130.613650] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6132.279199] Key type lgssc registered [ 6133.252138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6140.607283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6140.626043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.241917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.339372] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6152.676237] LustreError: 188304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.701445] LustreError: 188304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6154.940047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6155.372611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6159.810866] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6165.826976] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6170.291806] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6177.188859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6177.511216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.971074] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6187.772617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6190.663461] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6196.922687] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6209.171799] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6215.555917] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6221.106385] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6228.966922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.979215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.983474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6228.997670] Lustre: Skipped 1 previous similar message [ 6229.006855] Lustre: Skipped 1 previous similar message [ 6233.589867] LustreError: 191791:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6233.811790] Lustre: server umount lustre-OST0000 complete [ 6239.200586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.204347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6239.222675] Lustre: Skipped 1 previous similar message [ 6239.238533] Lustre: Skipped 1 previous similar message [ 6243.459612] LustreError: 191993:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6243.463384] LustreError: 191993:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6243.579426] Lustre: server umount lustre-MDT0000 complete [ 6244.330449] LustreError: 188304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.341783] LustreError: 188304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6247.066273] LustreError: 188283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560211 with bad export cookie 6699984000721709523 [ 6247.070173] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6247.075965] LustreError: 188283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6247.206884] LustreError: 192194:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6247.210520] LustreError: 192194:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6247.440490] Lustre: server umount lustre-MDT0001 complete [ 6257.661934] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6260.614838] Key type lgssc unregistered [ 6260.925155] LNet: 192726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6260.931807] LNetError: 192726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6260.948660] LNet: Removed LNI 192.168.202.133@tcp [ 6261.892467] Key type .llcrypt unregistered [ 6261.900900] Key type ._llcrypt unregistered [ 6287.673524] Key type ._llcrypt registered [ 6287.675565] Key type .llcrypt registered [ 6287.808461] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6289.035930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6289.086209] alg: No test for adler32 (adler32-zlib) [ 6290.149660] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 6290.575156] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6292.327361] Key type lgssc registered [ 6293.716435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.883124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6302.914278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.374909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.072682] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6313.963474] LustreError: 194118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.985546] LustreError: 194118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6317.592446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6318.021971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6321.822495] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6326.354516] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6329.461325] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.519601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6336.909892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.377263] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6349.160211] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6349.246939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6363.621637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6363.635756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.664484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.642820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.648643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.650819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.665761] Lustre: Skipped 1 previous similar message [ 6368.056133] LustreError: 196511:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6368.259163] Lustre: server umount lustre-OST0000 complete [ 6374.368369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6374.380984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.389462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.159937] LustreError: 196714:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6378.172870] LustreError: 196714:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6378.468694] Lustre: server umount lustre-MDT0000 complete [ 6380.018739] LustreError: 194113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.051193] LustreError: 194113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6383.391899] LustreError: 194098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560347 with bad export cookie 17214351899744250150 [ 6383.396439] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6383.397669] LustreError: 194098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6383.545803] LustreError: 196915:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6383.556969] LustreError: 196915:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6383.749484] Lustre: server umount lustre-MDT0001 complete [ 6395.837805] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6398.807592] Key type lgssc unregistered [ 6399.075814] LNet: 197447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6399.080492] LNetError: 197447:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6399.097686] LNet: Removed LNI 192.168.202.133@tcp [ 6400.054266] Key type .llcrypt unregistered [ 6400.056815] Key type ._llcrypt unregistered [ 6417.703364] Key type ._llcrypt registered [ 6417.707189] Key type .llcrypt registered [ 6417.813536] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6418.773829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6418.895075] alg: No test for adler32 (adler32-zlib) [ 6419.971501] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 6420.124936] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6421.783418] Key type lgssc registered [ 6422.797697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6429.832742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6429.852611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.364157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.220350] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6441.965883] LustreError: 198694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.977089] LustreError: 198694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6442.477850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6442.727895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6447.249855] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6453.069494] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6457.096529] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6463.490747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6463.933677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.474748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6474.646609] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6477.227027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6492.648049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6492.650042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6492.655446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6492.658594] Lustre: Skipped 1 previous similar message [ 6492.691748] LustreError: Skipped 1 previous similar message [ 6497.760968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6497.775624] Lustre: Skipped 2 previous similar messages [ 6498.399507] LustreError: 201088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6498.548993] Lustre: server umount lustre-OST0000 complete [ 6502.881258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.890022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6502.893425] Lustre: Skipped 1 previous similar message [ 6502.907479] Lustre: Skipped 1 previous similar message [ 6508.000932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6508.012974] Lustre: Skipped 1 previous similar message [ 6508.624765] LustreError: 201292:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6508.641447] LustreError: 201292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6508.754715] Lustre: server umount lustre-MDT0000 complete [ 6512.775856] LustreError: 198675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560476 with bad export cookie 5470542060307774054 [ 6512.777250] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6512.784398] LustreError: 198675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6512.901637] LustreError: 201494:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6512.909864] LustreError: 201494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6513.206579] Lustre: server umount lustre-MDT0001 complete [ 6524.983715] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6527.327559] Key type lgssc unregistered [ 6527.559860] LNet: 202027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6527.575396] LNetError: 202027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6527.592849] LNet: Removed LNI 192.168.202.133@tcp [ 6528.534058] Key type .llcrypt unregistered [ 6528.539037] Key type ._llcrypt unregistered [ 6548.365954] Key type ._llcrypt registered [ 6548.375311] Key type .llcrypt registered [ 6548.502727] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6549.751711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6549.811657] alg: No test for adler32 (adler32-zlib) [ 6550.935458] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 6551.129868] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6552.855524] Key type lgssc registered [ 6554.190799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6562.140457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6562.157770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.785108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6568.855708] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6573.029266] LustreError: 203274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.060929] LustreError: 203274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6578.148405] LustreError: 203274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.742041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6580.661782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6586.435497] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6594.021582] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6598.197682] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6606.559259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6607.064643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.115580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6613.318549] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6619.499541] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6631.118079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6631.278218] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6631.633822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.368563] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6646.332712] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6648.264901] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6660.007629] Lustre: Permanently deactivating lustre-OST0001 [ 6660.022948] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6665.914636] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6673.585394] Lustre: Permanently reactivating lustre-OST0001 [ 6673.607642] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6673.614542] Lustre: Skipped 2 previous similar messages [ 6673.724775] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.737138] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6673.745832] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6673.761584] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6680.641601] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.650598] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6680.658909] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6680.667112] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6685.683283] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6685.705563] Lustre: Skipped 1 previous similar message [ 6685.707173] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6686.275141] LustreError: 206716:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6686.383783] Lustre: server umount lustre-OST0001 complete [ 6690.791833] LustreError: 204870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.792461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.794599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.830727] LustreError: 204870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6690.854688] Lustre: Skipped 1 previous similar message [ 6695.395900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.404727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.416668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6695.914251] LustreError: 204871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.929326] LustreError: 204871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6696.578102] LustreError: 206917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6696.583731] LustreError: 206917:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6696.758249] Lustre: server umount lustre-OST0000 complete [ 6703.586666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6703.596501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.612955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6703.621859] Lustre: Skipped 1 previous similar message [ 6707.480505] LustreError: 207119:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6707.494652] LustreError: 207119:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6707.983634] Lustre: server umount lustre-MDT0000 complete [ 6711.264450] LustreError: 203275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.312098] LustreError: 203275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6712.433522] LustreError: 203255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768560676 with bad export cookie 16710591811774562207 [ 6712.434915] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.448565] LustreError: 203255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6712.621980] LustreError: 207320:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6712.628496] LustreError: 207320:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6712.865448] Lustre: server umount lustre-MDT0001 complete [ 6724.215575] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6727.295353] Key type lgssc unregistered [ 6727.645640] LNet: 207852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6727.658379] LNetError: 207852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6727.684821] LNet: Removed LNI 192.168.202.133@tcp [ 6728.472820] Key type .llcrypt unregistered [ 6728.474559] Key type ._llcrypt unregistered [ 6770.419969] Key type ._llcrypt registered [ 6770.428883] Key type .llcrypt registered [ 6770.530270] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6771.736751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6771.769491] alg: No test for adler32 (adler32-zlib) [ 6772.851909] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 6773.046482] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6774.767230] Key type lgssc registered [ 6775.795708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6782.557671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6782.583523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.827844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6783.847838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6784.031678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6787.785331] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6796.531082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6796.631660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6796.674066] Lustre: 210041:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6797.038703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6801.608715] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6807.127486] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6811.446515] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6818.009860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6818.196087] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6818.415073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6824.839180] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6826.193933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6829.605510] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6833.063489] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6840.691506] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6849.471415] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6855.915758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6864.917737] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6881.553471] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6881.557255] Lustre: Skipped 1 previous similar message [ 6918.478294] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6918.488478] Lustre: Skipped 3 previous similar messages [ 6987.952135] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6987.956623] Lustre: Skipped 7 previous similar messages [ 7021.024226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.030436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.047601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.050417] Lustre: Skipped 1 previous similar message [ 7021.067073] Lustre: Skipped 1 previous similar message [ 7026.154138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7026.161018] Lustre: Skipped 1 previous similar message [ 7026.685193] LustreError: 213577:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7026.864070] Lustre: server umount lustre-OST0000 complete [ 7032.812133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7032.819908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.837439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7036.385316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7036.389122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.692023] LustreError: 213780:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7037.698529] LustreError: 213780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7037.778852] Lustre: server umount lustre-MDT0000 complete [ 7041.834181] LustreError: 213023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561005 with bad export cookie 8035902086297239186 [ 7041.844670] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7041.990365] LustreError: 213981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7042.002293] LustreError: 213981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7042.575724] Lustre: server umount lustre-MDT0001 complete [ 7054.282450] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7058.998750] Key type lgssc unregistered [ 7059.555605] LNet: 214515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7059.567190] LNetError: 214515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7059.616798] LNet: Removed LNI 192.168.202.133@tcp [ 7060.570364] Key type .llcrypt unregistered [ 7060.572699] Key type ._llcrypt unregistered [ 7087.102493] Key type ._llcrypt registered [ 7087.105163] Key type .llcrypt registered [ 7087.245945] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7088.572429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7088.603217] alg: No test for adler32 (adler32-zlib) [ 7089.586947] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 7089.810158] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7091.503209] Key type lgssc registered [ 7092.553724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7099.622755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7099.641391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.992486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.390600] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7111.141671] LustreError: 215912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.160616] LustreError: 215912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7114.130409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7114.467840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7119.405405] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7125.170993] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7129.930416] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7136.996031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7137.553553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7143.667254] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7146.807570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7149.543527] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7171.554375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7171.571862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.594741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.577559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.594124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.602898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.625324] Lustre: Skipped 1 previous similar message [ 7174.525307] LustreError: 218313:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7174.674942] Lustre: server umount lustre-OST0000 complete [ 7181.280914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7181.295676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.328237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7185.382744] LustreError: 218514:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7185.397191] LustreError: 218514:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7185.723712] Lustre: server umount lustre-MDT0000 complete [ 7187.945063] LustreError: 215907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.970938] LustreError: 215907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7190.504344] LustreError: 215893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768561154 with bad export cookie 11431920908010931020 [ 7190.505579] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7190.516866] LustreError: 215893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7190.696401] LustreError: 218716:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7190.705552] LustreError: 218716:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7190.934206] Lustre: server umount lustre-MDT0001 complete [ 7202.785469] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7205.518908] Key type lgssc unregistered [ 7205.786235] LNet: 219248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7205.796511] LNetError: 219248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7205.810649] LNet: Removed LNI 192.168.202.133@tcp [ 7206.511476] Key type .llcrypt unregistered [ 7206.514068] Key type ._llcrypt unregistered [ 7227.235834] Key type ._llcrypt registered [ 7227.238726] Key type .llcrypt registered [ 7227.330202] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7228.567542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.770454] alg: No test for adler32 (adler32-zlib) [ 7229.922086] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 7230.116400] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7231.791097] Key type lgssc registered [ 7233.175065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7503.076575] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7506.217605] Key type lgssc unregistered [ 7506.478742] LNet: 221238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7506.484117] LNetError: 221238:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7506.500650] LNet: Removed LNI 192.168.202.133@tcp [ 7507.423929] Key type .llcrypt unregistered [ 7507.426088] Key type ._llcrypt unregistered [ 7536.617451] Key type ._llcrypt registered [ 7536.622526] Key type .llcrypt registered [ 7536.748096] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7557.656361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7559.405373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7559.449118] alg: No test for adler32 (adler32-zlib) [ 7560.657313] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 7561.102394] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7562.911217] Key type lgssc registered [ 7564.381224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7597.612486] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7343 sec ========== 06:05:59 (1768561559) [ 7599.474545] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:06:01 (1768561561) === [ 7602.597449] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:06:05 (1768561565) === [ 7647.058830] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7649.369531] Key type lgssc unregistered [ 7649.656915] LNet: 226422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7649.673384] LNetError: 226422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7649.695462] LNet: Removed LNI 192.168.202.133@tcp [ 7650.447957] Key type .llcrypt unregistered [ 7650.449943] Key type ._llcrypt unregistered