[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 579252891 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003233] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008741] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011158] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.024199] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030060] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032028] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034121] *** VALIDATE tmpfs *** [ 0.036355] *** VALIDATE proc *** [ 0.038143] *** VALIDATE cgroup *** [ 0.039011] *** VALIDATE cgroup2 *** [ 0.041200] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045030] Spectre V2 : User space: Vulnerable [ 0.046012] Speculative Store Bypass: Vulnerable [ 0.049000] debug: unmapping init [mem 0xffffffff97659000-0xffffffff97660fff] [ 0.050263] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051623] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052027] ... version: 2 [ 0.053014] ... bit width: 48 [ 0.054015] ... generic registers: 4 [ 0.055034] ... value mask: 0000ffffffffffff [ 0.056216] ... max period: 00007fffffffffff [ 0.057019] ... fixed-purpose events: 3 [ 0.058016] ... event mask: 000000070000000f [ 0.059321] rcu: Hierarchical SRCU implementation. [ 0.062252] smp: Bringing up secondary CPUs ... [ 0.063894] x86: Booting SMP configuration: [ 0.064243] .... node #0, CPUs: #1 #2 #3 [ 0.068026] smp: Brought up 1 node, 4 CPUs [ 0.070029] smpboot: Max logical packages: 1 [ 0.071014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.169021] node 0 deferred pages initialised in 96ms [ 0.174011] devtmpfs: initialized [ 0.175513] x86/mm: Memory block size: 128MB [ 0.180431] gcov: version magic: 0x41383552 [ 0.184278] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.185069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.186246] pinctrl core: initialized pinctrl subsystem [ 0.187160] [ 0.187551] ************************************************************* [ 0.188010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ** ** [ 0.190012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.191013] ** ** [ 0.192017] ** This means that this kernel is built to expose internal ** [ 0.193014] ** IOMMU data structures, which may compromise security on ** [ 0.194014] ** your system. ** [ 0.195018] ** ** [ 0.196014] ** If you see this message and you are not debugging the ** [ 0.197013] ** kernel, report this immediately to your vendor! ** [ 0.198009] ** ** [ 0.199010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200019] ************************************************************* [ 0.201970] NET: Registered protocol family 16 [ 0.204745] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218206] cpuidle: using governor menu [ 0.221853] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225619] PCI: Using configuration type 1 for base access [ 0.229135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.241288] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.245379] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.251037] cryptd: max_cpu_qlen set to 1000 [ 0.254230] ACPI: Added _OSI(Module Device) [ 0.257018] ACPI: Added _OSI(Processor Device) [ 0.259017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.268792] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.277552] ACPI: Interpreter enabled [ 0.279062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.280014] ACPI: Using IOAPIC for interrupt routing [ 0.283108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.287412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300781] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.304050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.308021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.313165] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.320604] acpiphp: Slot [2] registered [ 0.322161] acpiphp: Slot [5] registered [ 0.325141] acpiphp: Slot [6] registered [ 0.327177] acpiphp: Slot [7] registered [ 0.329142] acpiphp: Slot [8] registered [ 0.331243] acpiphp: Slot [9] registered [ 0.333125] acpiphp: Slot [10] registered [ 0.336140] acpiphp: Slot [3] registered [ 0.338125] acpiphp: Slot [4] registered [ 0.340140] acpiphp: Slot [11] registered [ 0.341121] acpiphp: Slot [12] registered [ 0.344117] acpiphp: Slot [13] registered [ 0.346111] acpiphp: Slot [14] registered [ 0.347211] acpiphp: Slot [15] registered [ 0.349137] acpiphp: Slot [16] registered [ 0.351232] acpiphp: Slot [17] registered [ 0.353645] acpiphp: Slot [18] registered [ 0.356100] acpiphp: Slot [19] registered [ 0.357114] acpiphp: Slot [20] registered [ 0.360091] acpiphp: Slot [21] registered [ 0.362351] acpiphp: Slot [22] registered [ 0.364113] acpiphp: Slot [23] registered [ 0.365096] acpiphp: Slot [24] registered [ 0.366176] acpiphp: Slot [25] registered [ 0.369115] acpiphp: Slot [26] registered [ 0.370096] acpiphp: Slot [27] registered [ 0.372125] acpiphp: Slot [28] registered [ 0.373306] acpiphp: Slot [29] registered [ 0.376116] acpiphp: Slot [30] registered [ 0.378161] acpiphp: Slot [31] registered [ 0.380069] PCI host bridge to bus 0000:00 [ 0.383025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.387020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.390266] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.394035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.398026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.402031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.405200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.409274] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.414990] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.429016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.435063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.445018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.448059] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.453400] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.457052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.461027] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.467017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.484015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.491018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.497240] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.508021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.516020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.536018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.550471] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.565018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.577033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.598019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.616115] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.627017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.635015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.661021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.676152] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.683018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.691017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.710018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.724062] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.735014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.748015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.779017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.794745] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.801023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.808025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.823000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.836409] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.839697] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.843483] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.846432] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.849510] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.855185] iommu: Default domain type: Passthrough [ 0.856000] SCSI subsystem initialized [ 0.856101] ACPI: bus type USB registered [ 0.858359] usbcore: registered new interface driver usbfs [ 0.861100] usbcore: registered new interface driver hub [ 0.864109] usbcore: registered new device driver usb [ 0.866152] pps_core: LinuxPPS API ver. 1 registered [ 0.868008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.870059] PTP clock support registered [ 0.873056] EDAC MC: Ver: 3.0.0 [ 0.875328] PCI: Using ACPI for IRQ routing [ 0.877948] NetLabel: Initializing [ 0.880020] NetLabel: domain hash size = 128 [ 0.881012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.884308] NetLabel: unlabeled traffic allowed by default [ 0.887154] vgaarb: loaded [ 0.889465] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.892019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.898579] clocksource: Switched to clocksource kvm-clock [ 1.043336] VFS: Disk quotas dquot_6.6.0 [ 1.045371] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.049430] *** VALIDATE ramfs *** [ 1.051050] *** VALIDATE hugetlbfs *** [ 1.053239] pnp: PnP ACPI init [ 1.057374] pnp: PnP ACPI: found 6 devices [ 1.076900] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.081147] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.084083] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.086675] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.089720] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.092973] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.097090] NET: Registered protocol family 2 [ 1.101036] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.109403] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.114553] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.120813] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.127102] TCP: Hash tables configured (established 65536 bind 65536) [ 1.132105] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.136871] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.141296] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.145414] NET: Registered protocol family 1 [ 1.150890] RPC: Registered named UNIX socket transport module. [ 1.153557] RPC: Registered udp transport module. [ 1.156507] RPC: Registered tcp transport module. [ 1.159069] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.162886] NET: Registered protocol family 44 [ 1.165631] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.168615] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.171590] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.174877] PCI: CLS 0 bytes, default 64 [ 1.177251] Unpacking initramfs... [ 3.047361] debug: unmapping init [mem 0xffff93a7bcc54000-0xffff93a7bffbffff] [ 3.053032] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.056911] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.060745] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.746274] Initialise system trusted keyrings [ 3.749138] Key type blacklist registered [ 3.751269] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.762888] zbud: loaded [ 3.767485] *** VALIDATE nfs *** [ 3.770119] *** VALIDATE nfs4 *** [ 3.773097] pstore: using deflate compression [ 3.777440] Platform Keyring initialized [ 3.909738] NET: Registered protocol family 38 [ 3.912669] Key type asymmetric registered [ 3.915430] Asymmetric key parser 'x509' registered [ 3.919299] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.924081] io scheduler mq-deadline registered [ 3.926489] io scheduler kyber registered [ 3.928341] io scheduler bfq registered [ 3.931780] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.936036] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.940968] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.945585] ACPI: Power Button [PWRF] [ 3.953458] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.962820] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.979775] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.991747] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.019814] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.050170] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.082309] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.090055] Non-volatile memory driver v1.3 [ 4.091771] Linux agpgart interface v0.103 [ 4.138375] virtio_blk virtio1: [vda] 139368 512-byte logical blocks (71.4 MB/68.1 MiB) [ 4.144052] vda: detected capacity change from 0 to 71356416 [ 4.163621] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.167774] vdb: detected capacity change from 0 to 1073741824 [ 4.199080] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.202499] vdc: detected capacity change from 0 to 2621440000 [ 4.221259] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.225044] vdd: detected capacity change from 0 to 2621440000 [ 4.245690] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.249904] vde: detected capacity change from 0 to 4294967296 [ 4.268953] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.273316] vdf: detected capacity change from 0 to 4294967296 [ 4.283115] libphy: Fixed MDIO Bus: probed [ 4.289512] usbcore: registered new interface driver usbserial_generic [ 4.292870] usbserial: USB Serial support registered for generic [ 4.296401] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.303555] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.306331] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.309484] mousedev: PS/2 mouse device common for all mice [ 4.313235] rtc_cmos 00:05: RTC can wake from S4 [ 4.316983] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.322858] rtc_cmos 00:05: registered as rtc0 [ 4.325452] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.326387] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.334388] intel_pstate: CPU model not supported [ 4.334393] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.344348] hid: raw HID events driver (C) Jiri Kosina [ 4.346888] usbcore: registered new interface driver usbhid [ 4.349650] usbhid: USB HID core driver [ 4.351758] drop_monitor: Initializing network drop monitor service [ 4.355331] Initializing XFRM netlink socket [ 4.358145] NET: Registered protocol family 10 [ 4.361679] Segment Routing with IPv6 [ 4.363666] NET: Registered protocol family 17 [ 4.366348] mpls_gso: MPLS GSO support [ 4.372800] RAS: Correctable Errors collector initialized. [ 4.376393] AVX version of gcm_enc/dec engaged. [ 4.379001] AES CTR mode by8 optimization enabled [ 4.476247] sched_clock: Marking stable (4476211137, 0)->(5717430558, -1241219421) [ 4.481241] registered taskstats version 1 [ 4.483938] Loading compiled-in X.509 certificates [ 4.486890] zswap: loaded using pool lzo/zbud [ 4.524732] Key type big_key registered [ 4.538477] Key type encrypted registered [ 4.540992] ima: No TPM chip found, activating TPM-bypass! [ 4.543898] ima: Allocated hash algorithm: sha1 [ 4.547346] ima: No architecture policies found [ 4.549814] evm: Initialising EVM extended attributes: [ 4.552460] evm: security.selinux [ 4.554235] evm: security.ima [ 4.555829] evm: security.capability [ 4.557531] evm: HMAC attrs: 0x1 [ 4.560939] rtc_cmos 00:05: setting system clock to 2026-02-10 00:44:57 UTC (1770684297) [ 4.568798] debug: unmapping init [mem 0xffffffff98603000-0xffffffff987fffff] [ 4.573463] debug: unmapping init [mem 0xffffffff97382000-0xffffffff97658fff] [ 4.586033] Write protecting the kernel read-only data: 28672k [ 4.590725] debug: unmapping init [mem 0xffffffff95a03000-0xffffffff95bfffff] [ 4.594784] debug: unmapping init [mem 0xffffffff96314000-0xffffffff963fffff] [ 4.645344] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.657289] systemd[1]: Detected virtualization kvm. [ 4.659484] systemd[1]: Detected architecture x86-64. [ 4.662214] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.691371] systemd[1]: No hostname configured. [ 4.694384] systemd[1]: Set hostname to . [ 4.697530] random: systemd: uninitialized urandom read (16 bytes read) [ 4.701450] systemd[1]: Initializing machine ID from random generator. [ 4.763482] random: ln: uninitialized urandom read (6 bytes read) [ 4.876309] random: systemd: uninitialized urandom read (16 bytes read) [ 4.880757] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.887524] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.892851] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.735348] device-mapper: uevent: version 1.0.3 [ 5.738390] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.687525] virtio_net virtio0 ens2: renamed from eth0 [ 6.704377] random: fast init done [ 6.755478] scsi host0: ata_piix [ 6.766293] scsi host1: ata_piix [ 6.784937] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.789412] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.002524] dracut-initqueue[579]: RTNETLINK answers: File exists [ 11.068532] random: crng init done [ 11.070190] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.357834] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.736601] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.032482] SELinux: Disabled at runtime. [ 14.104919] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.116608] systemd[1]: Detected virtualization kvm. [ 14.119064] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.793456] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.798124] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.803614] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.809810] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.814434] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.824564] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.834839] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ 15.018088] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 15.481362] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.869477] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.920178] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.053376] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.079551] EDAC sbridge: Ver: 1.1.2 [ 17.742568] Key type dns_resolver registered [ 18.130447] NFS: Registering the id_resolver key type [ 18.133640] Key type id_resolver registered [ 18.137023] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg128-server login: [ 49.963937] libcfs: loading out-of-tree module taints kernel. [ 49.990862] Key type ._llcrypt registered [ 49.992686] Key type .llcrypt registered [ 50.062997] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 69.334387] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 72.229172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.277922] alg: No test for adler32 (adler32-zlib) [ 73.341183] hrtimer: interrupt took 6168560 ns [ 74.435855] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 76.219903] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 78.265139] Key type lgssc registered [ 81.998542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.242047] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.184163] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 192.444146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.505582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.956124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.015457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.218566] Lustre: lustre-MDT0000: new disk, initializing [ 194.398542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.442354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.058738] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.550083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 220.786571] Lustre: 6518:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.861844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.874377] Lustre: Skipped 1 previous similar message [ 221.080418] Lustre: lustre-MDT0001: new disk, initializing [ 221.199813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.254425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.272076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.034973] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.712333] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.465346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 248.869381] Lustre: lustre-OST0000: new disk, initializing [ 248.878321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.903347] Lustre: 8423:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.988160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.105429] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.135211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.158482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.249038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 278.308665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 278.503932] Lustre: lustre-OST0001: new disk, initializing [ 278.512619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.521203] Lustre: 9479:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 278.611241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.345036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.392973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.548170] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 288.171788] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 304.652872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.818713] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 330.081621] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing check_logdir /tmp/testlogs/ [ 337.895612] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing yml_node [ 345.065929] Lustre: DEBUG MARKER: Client: 2.17.50.65 [ 349.419724] Lustre: DEBUG MARKER: MDS: 2.17.50.65 [ 353.624651] Lustre: DEBUG MARKER: OSS: 2.17.50.65 [ 356.561856] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Feb 9 19:50:45 EST 2026 [ 383.529654] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 386.047996] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 405.480258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 405.486840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.517215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 408.553637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 408.572383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.608618] Lustre: Skipped 2 previous similar messages [ 413.672599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 413.682620] Lustre: Skipped 6 previous similar messages [ 418.787201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.802135] Lustre: Skipped 2 previous similar messages [ 419.296190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 419.528601] LustreError: 12228:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 419.655947] Lustre: server umount lustre-MDT0000 complete [ 425.654417] LustreError: 7449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684718 with bad export cookie 7761978983678364573 [ 425.658979] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 425.683350] LustreError: 7449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 429.039192] LustreError: 7712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.042495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.065053] LustreError: 7712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 429.135066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 430.049477] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.079117] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 431.939265] LustreError: 12431:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 431.961463] LustreError: 12431:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 432.307640] Lustre: server umount lustre-MDT0001 complete [ 438.562733] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 438.579219] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 438.849898] Lustre: server umount lustre-OST0000 complete [ 444.198960] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 444.201975] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 444.472909] Lustre: server umount lustre-OST0001 complete [ 455.323163] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 475.058512] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 537.745133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.354370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 538.468643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 538.725440] Lustre: lustre-MDT0000: new disk, initializing [ 538.910433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.981530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 547.056267] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 564.556856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.810130] Lustre: 15034:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 564.900717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 564.905480] Lustre: Skipped 1 previous similar message [ 565.038427] Lustre: lustre-MDT0001: new disk, initializing [ 565.248387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 565.331614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 565.358803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 573.470373] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 586.969645] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 593.195799] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 603.297129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 603.696626] Lustre: lustre-OST0000: new disk, initializing [ 603.709476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 603.730624] Lustre: 16779:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 603.867396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.023979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 605.049599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 605.326390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 614.123451] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 624.757325] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 631.022705] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 631.399104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 638.570575] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 638.992571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 646.130306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 646.137980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.166576] Lustre: Skipped 1 previous similar message [ 646.180383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.204280] Lustre: Skipped 4 previous similar messages [ 647.640658] LustreError: 17972:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 647.656306] LustreError: 17972:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 647.910069] Lustre: server umount lustre-OST0000 complete [ 656.354766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.378239] Lustre: Skipped 2 previous similar messages [ 659.998439] LustreError: 18175:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 660.017539] LustreError: 18175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 660.145776] Lustre: server umount lustre-MDT0000 complete [ 661.488331] LustreError: 15045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.528121] LustreError: 15045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 666.254162] LustreError: 16789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684959 with bad export cookie 7761978983678366106 [ 666.256141] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.267816] LustreError: 16789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 666.882806] Lustre: server umount lustre-MDT0001 complete [ 673.081588] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:56:02 (1770684962) [ 683.856295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.682451] LustreError: 18904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.842951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.157235] LustreError: 18904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.844117] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 699.386524] LustreError: 18903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.435306] LustreError: 18903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 704.725301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 713.499442] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 722.686040] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 728.809824] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 740.552291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 751.704618] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 766.766514] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 785.379613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 785.395018] LustreError: Skipped 1 previous similar message [ 785.426698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.436909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 785.457467] Lustre: Skipped 1 previous similar message [ 785.490327] Lustre: Skipped 3 previous similar messages [ 788.754156] LustreError: 21174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 788.769544] LustreError: 21174:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 789.106751] Lustre: server umount lustre-OST0000 complete [ 801.957129] Lustre: server umount lustre-MDT0000 complete [ 805.858068] LustreError: 19616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.867629] LustreError: 19616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 808.013920] LustreError: 18884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685100 with bad export cookie 7761978983678366981 [ 808.045029] LustreError: 18884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 808.049510] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.976503] Lustre: server umount lustre-MDT0001 complete [ 828.039323] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 835.259673] Key type lgssc unregistered [ 836.100449] LNet: 22113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.116184] LNetError: 22113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.190775] LNet: Removed LNI 192.168.201.128@tcp [ 838.637448] Key type .llcrypt unregistered [ 838.653450] Key type ._llcrypt unregistered [ 875.198600] Key type ._llcrypt registered [ 875.201549] Key type .llcrypt registered [ 875.340510] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 876.642074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.666054] alg: No test for adler32 (adler32-zlib) [ 878.023874] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 878.304879] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 879.992207] Key type lgssc registered [ 881.551166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.791778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.823220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.794432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.699774] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 904.170451] LustreError: 23507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.218446] LustreError: 23507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 909.287928] LustreError: 23507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.409255] LustreError: 23507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.711288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 916.425344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 923.304174] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 932.445410] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 939.977412] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 953.987687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 954.728257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.346857] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 965.005746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 974.781533] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1000.939881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.967044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.975248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.004902] Lustre: Skipped 1 previous similar message [ 1001.037767] Lustre: Skipped 1 previous similar message [ 1006.064189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.076647] Lustre: Skipped 1 previous similar message [ 1006.220282] LustreError: 26213:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1006.802964] Lustre: server umount lustre-OST0000 complete [ 1013.731210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1013.754782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.783356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.294561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.306194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.377926] Lustre: Skipped 1 previous similar message [ 1019.404157] LustreError: 26415:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1019.434119] LustreError: 26415:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1019.704740] Lustre: server umount lustre-MDT0000 complete [ 1021.413474] LustreError: 23508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.452437] LustreError: 23508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1026.535325] LustreError: 23503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.569420] LustreError: 23503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1027.001124] LustreError: 23489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685319 with bad export cookie 16452130406025802860 [ 1027.005923] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.013267] LustreError: 23489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1027.217822] LustreError: 26618:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1027.236404] LustreError: 26618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1027.509851] Lustre: server umount lustre-MDT0001 complete [ 1049.000710] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1053.377596] Key type lgssc unregistered [ 1053.823160] LNet: 27174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.838197] LNetError: 27174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.867361] LNet: Removed LNI 192.168.201.128@tcp [ 1055.049439] Key type .llcrypt unregistered [ 1055.052321] Key type ._llcrypt unregistered [ 1094.288608] Key type ._llcrypt registered [ 1094.292714] Key type .llcrypt registered [ 1094.545269] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1097.121344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1097.142236] alg: No test for adler32 (adler32-zlib) [ 1098.602707] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1099.341655] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1101.144503] Key type lgssc registered [ 1103.054445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.742220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.771232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.589305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.156610] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1127.908306] LustreError: 28579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.932808] LustreError: 28579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1133.035427] LustreError: 28579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.149868] LustreError: 28579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.677781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1140.460511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1147.375490] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1156.923883] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1164.200209] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1187.160871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1187.632166] LustreError: 30524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.841238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.971123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1196.226130] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1206.229643] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.652582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1223.666561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.716707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1225.382311] LustreError: 31194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1225.768345] Lustre: server umount lustre-OST0000 complete [ 1231.847235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.855685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.876274] Lustre: Skipped 1 previous similar message [ 1236.968484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.984199] Lustre: Skipped 2 previous similar messages [ 1242.080452] Lustre: 27892:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770685519/real 1770685519] req@000000002a1d13b2 x1856698205638784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770685535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1242.082419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.089121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.122589] Lustre: 27892:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1246.176141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1246.332704] LustreError: 31395:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1246.348093] LustreError: 31395:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1246.607302] Lustre: server umount lustre-MDT0000 complete [ 1247.202881] Lustre: 27891:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770685524/real 1770685524] req@00000000b4bd9217 x1856698205640448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770685540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1247.202887] LustreError: 28579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.325131] Lustre: 27889:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770685529/real 1770685529] req@0000000052783bfb x1856698205641216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770685545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1252.465750] LustreError: 28559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685545 with bad export cookie 5054343328188878684 [ 1252.483655] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.500084] LustreError: 28559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1252.708707] LustreError: 31599:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1252.716852] LustreError: 31599:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1253.300639] Lustre: server umount lustre-MDT0001 complete [ 1272.074643] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1277.283242] Key type lgssc unregistered [ 1277.753978] LNet: 32146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.762552] LNetError: 32146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.782174] LNet: Removed LNI 192.168.201.128@tcp [ 1278.819929] Key type .llcrypt unregistered [ 1278.825411] Key type ._llcrypt unregistered [ 1316.797574] Key type ._llcrypt registered [ 1316.802443] Key type .llcrypt registered [ 1316.929980] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1319.067397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1319.195507] alg: No test for adler32 (adler32-zlib) [ 1320.304263] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1320.553800] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1322.258762] Key type lgssc registered [ 1323.989452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.741080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1336.794075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.812994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1345.195172] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1348.071366] LustreError: 33542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.116189] LustreError: 33542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1353.196313] LustreError: 33542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.866956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1357.384816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1363.824670] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1374.975379] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.058558] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1394.714249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1395.185427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.929628] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1408.522700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1414.621708] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1433.059385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1433.076462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.094067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.088534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.092566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.125120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.745414] LustreError: 35813:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1436.094817] Lustre: server umount lustre-OST0000 complete [ 1444.324709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1444.342367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.348878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.386437] Lustre: Skipped 1 previous similar message [ 1444.412161] Lustre: Skipped 2 previous similar messages [ 1448.643180] LustreError: 36015:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1448.660457] LustreError: 36015:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1448.914228] Lustre: server umount lustre-MDT0000 complete [ 1449.443973] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.468354] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1454.565979] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.589180] LustreError: 33543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1455.092707] LustreError: 33523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685748 with bad export cookie 9604144139948274033 [ 1455.116902] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.123877] LustreError: 33523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1455.371531] LustreError: 36217:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1455.380541] LustreError: 36217:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1455.651811] Lustre: server umount lustre-MDT0001 complete [ 1472.516811] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1476.421696] Key type lgssc unregistered [ 1476.861954] LNet: 36749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.878026] LNetError: 36749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.897058] LNet: Removed LNI 192.168.201.128@tcp [ 1478.135954] Key type .llcrypt unregistered [ 1478.143997] Key type ._llcrypt unregistered [ 1510.052905] Key type ._llcrypt registered [ 1510.057253] Key type .llcrypt registered [ 1510.223227] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1511.834389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1512.062133] alg: No test for adler32 (adler32-zlib) [ 1513.237602] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1513.529815] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1515.264309] Key type lgssc registered [ 1516.880977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1528.591841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1528.618593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.421103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.056654] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1540.580791] LustreError: 38142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.616744] LustreError: 38142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1545.703385] LustreError: 38142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.500627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1548.908577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1554.231275] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1562.381705] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1567.568774] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1576.252378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1576.617321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.824699] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1592.270699] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1600.126673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1610.724797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.740116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1610.754567] Lustre: Skipped 1 previous similar message [ 1610.755523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1610.814373] Lustre: Skipped 1 previous similar message [ 1614.592152] Lustre: lustre-OST0000: Not available for connect from 192.168.201.28@tcp (stopping) [ 1615.848189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.871833] Lustre: Skipped 1 previous similar message [ 1616.274940] LustreError: 40410:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1616.554046] Lustre: server umount lustre-OST0000 complete [ 1645.538481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.558132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.580629] Lustre: Skipped 1 previous similar message [ 1647.521224] LustreError: 40789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1647.528894] LustreError: 40789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1647.868060] Lustre: server umount lustre-MDT0000 complete [ 1650.663870] LustreError: 39610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.704386] LustreError: 39610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1652.980171] LustreError: 39762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685945 with bad export cookie 3212308722776757116 [ 1652.981671] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.989121] LustreError: 39762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1659.195570] LustreError: 40991:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1659.208189] LustreError: 40991:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1659.453986] Lustre: server umount lustre-MDT0001 complete [ 1674.657745] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1678.639575] Key type lgssc unregistered [ 1679.053354] LNet: 41523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.070268] LNetError: 41523:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.105194] LNet: Removed LNI 192.168.201.128@tcp [ 1680.365489] Key type .llcrypt unregistered [ 1680.368424] Key type ._llcrypt unregistered [ 1710.367661] Key type ._llcrypt registered [ 1710.372292] Key type .llcrypt registered [ 1710.496728] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1711.566847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.598062] alg: No test for adler32 (adler32-zlib) [ 1712.713682] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1712.981866] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1714.728310] Key type lgssc registered [ 1715.900671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.504810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.565927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.512944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.945643] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1737.701746] LustreError: 42918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.733610] LustreError: 42918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1742.757090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1742.819768] LustreError: 42918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.137729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.088554] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1754.794166] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1759.038239] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1767.711481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.909192] LustreError: 44514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.204755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.283342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1774.852495] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1781.540572] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1787.874839] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.880130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.910109] Lustre: Skipped 1 previous similar message [ 1792.998516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.023383] Lustre: Skipped 5 previous similar messages [ 1793.668135] LustreError: 45162:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1793.791196] Lustre: server umount lustre-MDT0000 complete [ 1798.120120] LustreError: 42913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.155037] LustreError: 42913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1799.148259] LustreError: 42897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686092 with bad export cookie 15469401555945951899 [ 1799.165774] LustreError: 42897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1799.168191] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.237893] LustreError: 42913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1803.240415] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.285185] LustreError: 42913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1803.320651] Lustre: Skipped 1 previous similar message [ 1803.327744] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1803.331343] Lustre: Skipped 1 previous similar message [ 1805.405142] LustreError: 45366:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1805.413186] LustreError: 45366:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1805.599532] Lustre: server umount lustre-MDT0001 complete [ 1827.046206] LustreError: 45589:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1827.061445] LustreError: 45589:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1827.215503] Lustre: server umount lustre-OST0000 complete [ 1846.818467] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1849.638817] Key type lgssc unregistered [ 1849.936702] LNet: 46428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.950432] LNetError: 46428:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.977268] LNet: Removed LNI 192.168.201.128@tcp [ 1850.779891] Key type .llcrypt unregistered [ 1850.782161] Key type ._llcrypt unregistered [ 1877.153106] Key type ._llcrypt registered [ 1877.154743] Key type .llcrypt registered [ 1877.272374] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1878.469302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.501328] alg: No test for adler32 (adler32-zlib) [ 1879.555618] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1879.770867] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1881.432232] Key type lgssc registered [ 1882.675732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1892.116920] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1892.129867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1907.682925] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1912.801351] LustreError: 47812:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 1939.552203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.718861] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1955.275926] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2123.519798] LustreError: 48520:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2123.681321] Lustre: server umount lustre-OST0000 complete [ 2143.684709] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2146.971883] Key type lgssc unregistered [ 2147.249203] LNet: 49358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.263418] LNetError: 49358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.289880] LNet: Removed LNI 192.168.201.128@tcp [ 2148.202978] Key type .llcrypt unregistered [ 2148.208061] Key type ._llcrypt unregistered [ 2175.607899] Key type ._llcrypt registered [ 2175.615889] Key type .llcrypt registered [ 2175.804992] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2177.210555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.237200] alg: No test for adler32 (adler32-zlib) [ 2178.641653] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2179.041307] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2180.800405] Key type lgssc registered [ 2182.467531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2192.112205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2192.133230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.739743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.023096] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2203.112331] LustreError: 50752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.162512] LustreError: 50752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2208.229406] LustreError: 50752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.406582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2209.989383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.482087] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2222.899145] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2227.894686] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2236.494099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2237.048908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.112667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2244.994834] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2252.535546] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2259.954230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2259.963605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.972880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2261.579223] LustreError: 52996:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2261.743913] Lustre: server umount lustre-OST0000 complete [ 2266.082385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2266.090828] LustreError: Skipped 1 previous similar message [ 2266.094340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.117561] Lustre: Skipped 1 previous similar message [ 2266.133089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.138281] Lustre: Skipped 1 previous similar message [ 2270.180707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.183204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.213651] Lustre: Skipped 1 previous similar message [ 2272.077910] LustreError: 53199:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2272.095835] LustreError: 53199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2272.410919] Lustre: server umount lustre-MDT0000 complete [ 2275.297768] LustreError: 51463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.350348] LustreError: 51463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2276.914106] LustreError: 52373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686569 with bad export cookie 3858538577228230039 [ 2276.916423] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.924583] LustreError: 52373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2277.050923] LustreError: 53400:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2277.061674] LustreError: 53400:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2277.388867] Lustre: server umount lustre-MDT0001 complete [ 2289.858943] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2293.092632] Key type lgssc unregistered [ 2293.414966] LNet: 53933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.422287] LNetError: 53933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.441884] LNet: Removed LNI 192.168.201.128@tcp [ 2294.312738] Key type .llcrypt unregistered [ 2294.315787] Key type ._llcrypt unregistered [ 2321.486751] Key type ._llcrypt registered [ 2321.489522] Key type .llcrypt registered [ 2321.618931] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2323.224368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.316551] alg: No test for adler32 (adler32-zlib) [ 2324.433175] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2324.688920] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2326.457742] Key type lgssc registered [ 2327.787210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2336.941954] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2336.960911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2352.544441] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2357.667579] LustreError: 55300:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 2384.416290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.209917] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2398.849369] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2406.575724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.370178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.529524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2412.534175] LustreError: 55325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.594600] LustreError: 55325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2413.452763] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2417.644216] LustreError: 55325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.701478] LustreError: 56222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.719208] LustreError: 56222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2423.876626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2424.242370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2429.444868] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2437.449296] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2442.268976] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2448.353854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.363183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.384599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.892505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.904596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.907905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.938053] Lustre: Skipped 1 previous similar message [ 2451.038423] LustreError: 57573:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2451.210545] Lustre: server umount lustre-OST0000 complete [ 2479.589942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.596322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.622044] Lustre: Skipped 1 previous similar message [ 2479.648325] Lustre: Skipped 1 previous similar message [ 2483.325419] LustreError: 57952:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2483.328954] LustreError: 57952:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2483.467702] Lustre: server umount lustre-MDT0000 complete [ 2484.707477] LustreError: 56221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.728397] LustreError: 56221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2488.509210] LustreError: 55307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686781 with bad export cookie 13429676364508402936 [ 2488.526343] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.914787] LustreError: 58154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2494.929058] LustreError: 58154:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2495.148594] Lustre: server umount lustre-MDT0001 complete [ 2507.937555] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2511.077923] Key type lgssc unregistered [ 2511.353878] LNet: 58689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.365174] LNetError: 58689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.390607] LNet: Removed LNI 192.168.201.128@tcp [ 2512.158984] Key type .llcrypt unregistered [ 2512.161350] Key type ._llcrypt unregistered [ 2539.507202] Key type ._llcrypt registered [ 2539.511228] Key type .llcrypt registered [ 2539.635488] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2541.059994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2541.174975] alg: No test for adler32 (adler32-zlib) [ 2542.246396] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2542.534300] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2544.273981] Key type lgssc registered [ 2545.401188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2553.688000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2553.754911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.317480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.808351] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2565.610302] LustreError: 60083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.637818] LustreError: 60083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2570.734238] LustreError: 60083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.741227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2572.424859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2577.264609] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2584.161439] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2589.206992] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2596.707124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2597.325242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.381314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2605.293471] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2611.314613] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.100103] LustreError: 62350:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2631.140246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.141889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.160030] LustreError: 61679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.172152] LustreError: 61679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2631.233622] Lustre: server umount lustre-OST0000 complete [ 2636.258967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.266039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.273326] Lustre: Skipped 2 previous similar messages [ 2636.288972] Lustre: Skipped 1 previous similar message [ 2641.210344] LustreError: 62551:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2641.230044] LustreError: 62551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2641.327425] Lustre: server umount lustre-MDT0000 complete [ 2641.382726] LustreError: 60084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.403183] LustreError: 60084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2645.708093] LustreError: 60063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686938 with bad export cookie 17337203217624021972 [ 2645.715830] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.720121] LustreError: 60063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2645.923151] LustreError: 62753:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2645.936808] LustreError: 62753:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2646.184851] Lustre: server umount lustre-MDT0001 complete [ 2656.994532] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2659.675298] Key type lgssc unregistered [ 2660.074532] LNet: 63286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.084177] LNetError: 63286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2661.158172] LNet: Removed LNI 192.168.201.128@tcp [ 2662.037849] Key type .llcrypt unregistered [ 2662.040339] Key type ._llcrypt unregistered [ 2687.175140] Key type ._llcrypt registered [ 2687.182192] Key type .llcrypt registered [ 2687.299707] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2688.331695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.344178] alg: No test for adler32 (adler32-zlib) [ 2689.486900] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2689.694721] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2691.394251] Key type lgssc registered [ 2692.547969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2700.528244] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:29:51 (1770686991) [ 2706.908210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2706.926413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.301591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.014624] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2718.690194] LustreError: 64969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.707299] LustreError: 64969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2720.849865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2721.408227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.983769] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2731.348493] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2734.584538] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2740.712123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2741.051866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.699204] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2753.391991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2753.657651] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2760.231224] Lustre: Failing over lustre-MDT0000 [ 2760.398424] LustreError: 67211:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2760.524617] Lustre: server umount lustre-MDT0000 complete [ 2761.489132] LustreError: 67124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.212243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.219676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.752745] LustreError: 64969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.754319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.777738] LustreError: 64969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2763.806908] Lustre: Skipped 1 previous similar message [ 2766.577946] LustreError: 67124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.592431] LustreError: 67124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2770.992905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.161680] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.320814] Lustre: *** cfs_fail_loc=135, val=0*** [ 2771.441908] LustreError: 67720:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2771.450148] LustreError: 67720:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.128@tcp: cfg command failed: rc = -2 [ 2771.456748] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2771.456748] [ 2771.464593] LustreError: MGC192.168.201.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2771.474743] LustreError: 67709:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2771.482123] LustreError: 67709:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 2771.490527] LustreError: 67709:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2771.502291] LustreError: 67709:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2771.540624] Lustre: server umount lustre-MDT0000 complete [ 2771.543658] LustreError: 67709:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2771.682675] LustreError: 64965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.699741] LustreError: 64965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2779.329860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.441676] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.454459] LustreError: 68065:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.201.128@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000495800bd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2779.475977] LustreError: 64961:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2779.726933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.797702] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2782.948550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2784.263893] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2784.768875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2784.786036] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2784.846646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2792.421774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2792.443190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.468167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2795.005080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.006506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2795.043123] Lustre: Skipped 1 previous similar message [ 2800.108310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2800.127544] Lustre: Skipped 1 previous similar message [ 2805.220352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.228450] Lustre: Skipped 1 previous similar message [ 2806.240895] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2806.298134] LustreError: 68528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2806.301673] LustreError: 68528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2806.392536] Lustre: server umount lustre-OST0000 complete [ 2810.337898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.341093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.347323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.360966] Lustre: Skipped 1 previous similar message [ 2816.402741] LustreError: 68733:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2816.410936] LustreError: 68733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2816.460165] Lustre: server umount lustre-MDT0000 complete [ 2820.004879] LustreError: 64949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687112 with bad export cookie 1107061269769190745 [ 2820.012312] LustreError: 64949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2820.012682] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2820.152941] LustreError: 68934:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2820.163772] LustreError: 68934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2820.350861] Lustre: server umount lustre-MDT0001 complete [ 2829.899240] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2832.594636] Key type lgssc unregistered [ 2832.894079] LNet: 69467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2832.906482] LNetError: 69467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2832.923586] LNet: Removed LNI 192.168.201.128@tcp [ 2833.604487] Key type .llcrypt unregistered [ 2833.606082] Key type ._llcrypt unregistered [ 2852.395789] Key type ._llcrypt registered [ 2852.399116] Key type .llcrypt registered [ 2852.527697] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2853.299559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2853.392094] alg: No test for adler32 (adler32-zlib) [ 2854.390188] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2854.553259] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2856.233415] Key type lgssc registered [ 2857.183809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.937235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2863.976637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.468396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.969030] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2875.875117] LustreError: 70842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.900170] LustreError: 70842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2876.750872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2877.081775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2881.003393] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2886.331567] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.677852] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2895.168846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2895.447881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.519658] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2903.864549] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2910.842886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2919.061558] Lustre: Failing over lustre-MDT0000 [ 2919.244151] LustreError: 73108:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2919.411366] Lustre: server umount lustre-MDT0000 complete [ 2919.702186] LustreError: 72996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.453125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.463695] LustreError: 70838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.467318] Lustre: Skipped 2 previous similar messages [ 2921.502231] LustreError: 70838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2924.771547] LustreError: 70837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.985140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.150211] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.294940] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2928.394269] LustreError: 73616:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2928.402278] LustreError: 73616:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.128@tcp: cfg command failed: rc = -9 [ 2928.414160] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2928.414160] [ 2928.422102] LustreError: MGC192.168.201.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2928.436378] LustreError: 73605:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2928.442947] LustreError: 73605:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -9 [ 2928.450591] LustreError: 73605:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2928.455852] LustreError: 73605:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2928.472429] Lustre: server umount lustre-MDT0000 complete [ 2928.475357] LustreError: 73605:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2929.890839] LustreError: 70838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.915368] LustreError: 70838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2933.749762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.845921] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.999641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.038861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2935.005082] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2936.563560] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2939.371610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2939.392953] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2939.431468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2942.945801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2942.956269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.974349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.483737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.485339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.501764] Lustre: Skipped 1 previous similar message [ 2948.851109] LustreError: 74425:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2948.854182] LustreError: 74425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2948.936756] Lustre: server umount lustre-OST0000 complete [ 2954.722176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.727854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2954.732403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.743993] Lustre: Skipped 1 previous similar message [ 2958.076231] LustreError: 74626:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2958.086308] LustreError: 74626:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2958.194273] Lustre: server umount lustre-MDT0000 complete [ 2959.846031] LustreError: 70842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.877152] LustreError: 70842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2961.721792] LustreError: 70823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687254 with bad export cookie 3373799121743162673 [ 2961.733610] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.843603] LustreError: 74828:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2961.852221] LustreError: 74828:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2961.954689] Lustre: server umount lustre-MDT0001 complete [ 2970.460512] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2972.521290] Key type lgssc unregistered [ 2972.754523] LNet: 75360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2972.762434] LNetError: 75360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2972.780455] LNet: Removed LNI 192.168.201.128@tcp [ 2973.353654] Key type .llcrypt unregistered [ 2973.356264] Key type ._llcrypt unregistered [ 2983.441544] Key type ._llcrypt registered [ 2983.444139] Key type .llcrypt registered [ 2983.538302] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2983.815540] Key type .llcrypt unregistered [ 2983.820351] Key type ._llcrypt unregistered [ 3002.163472] Key type ._llcrypt registered [ 3002.169371] Key type .llcrypt registered [ 3002.276348] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3003.003453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3003.021854] alg: No test for adler32 (adler32-zlib) [ 3004.016706] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3004.214426] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3005.856238] Key type lgssc registered [ 3006.881097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3013.247454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3013.262867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.624590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.944395] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3024.865273] LustreError: 77200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.870358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3024.880768] LustreError: 77200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3025.190771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3028.678543] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3032.600723] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3035.509842] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3041.196388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3041.576680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.895861] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3051.379742] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.019180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3069.414037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.419441] Lustre: Skipped 1 previous similar message [ 3069.420214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.433154] Lustre: Skipped 1 previous similar message [ 3074.530568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.534966] Lustre: Skipped 1 previous similar message [ 3074.948937] LustreError: 79462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3075.076420] Lustre: server umount lustre-OST0000 complete [ 3079.650858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.658728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.660062] Lustre: Skipped 1 previous similar message [ 3079.672115] Lustre: Skipped 1 previous similar message [ 3084.345418] LustreError: 79664:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3084.357111] LustreError: 79664:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3084.447931] Lustre: server umount lustre-MDT0000 complete [ 3084.771267] LustreError: 77200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.794235] LustreError: 77200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3087.554473] LustreError: 77180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687380 with bad export cookie 741161329966320399 [ 3087.568742] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.574698] LustreError: 77180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3087.722426] LustreError: 79865:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3087.732494] LustreError: 79865:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3087.844724] Lustre: server umount lustre-MDT0001 complete [ 3096.467141] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3098.529841] Key type lgssc unregistered [ 3098.762200] LNet: 80398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3098.767799] LNetError: 80398:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3098.785509] LNet: Removed LNI 192.168.201.128@tcp [ 3099.394195] Key type .llcrypt unregistered [ 3099.397218] Key type ._llcrypt unregistered [ 3118.611178] Key type ._llcrypt registered [ 3118.613360] Key type .llcrypt registered [ 3118.695272] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3119.574823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3119.594715] alg: No test for adler32 (adler32-zlib) [ 3120.629024] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3120.810101] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3122.424183] Key type lgssc registered [ 3123.449600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3129.467351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3129.499198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.920913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.706972] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3140.068247] LustreError: 81768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.082088] LustreError: 81768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3140.791384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3141.084159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3144.005502] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3148.396214] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3151.340438] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3156.611503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3156.894325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.372717] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3166.993898] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3175.348299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3185.638651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.640511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3185.662295] Lustre: Skipped 1 previous similar message [ 3185.662667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3185.676943] Lustre: Skipped 1 previous similar message [ 3190.757329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.764336] Lustre: Skipped 1 previous similar message [ 3195.879309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.884944] Lustre: Skipped 1 previous similar message [ 3197.408133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3197.476538] LustreError: 84032:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3197.610742] Lustre: server umount lustre-OST0000 complete [ 3200.998497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.025809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.035122] Lustre: Skipped 1 previous similar message [ 3206.114691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.121605] Lustre: Skipped 1 previous similar message [ 3215.329475] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3215.419381] LustreError: 84234:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3215.423813] LustreError: 84234:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3215.498458] Lustre: server umount lustre-MDT0000 complete [ 3216.357453] LustreError: 82479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.376314] LustreError: 82479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3218.234179] LustreError: 81748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687511 with bad export cookie 4982647951626368568 [ 3218.238854] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3218.249800] LustreError: 81748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3218.356854] LustreError: 84437:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3218.365725] LustreError: 84437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3218.500366] Lustre: server umount lustre-MDT0001 complete [ 3227.602380] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3230.273447] Key type lgssc unregistered [ 3230.548696] LNet: 84969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.559603] LNetError: 84969:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.574729] LNet: Removed LNI 192.168.201.128@tcp [ 3231.221686] Key type .llcrypt unregistered [ 3231.224864] Key type ._llcrypt unregistered [ 3250.260651] Key type ._llcrypt registered [ 3250.262939] Key type .llcrypt registered [ 3250.382807] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3251.117230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3251.166088] alg: No test for adler32 (adler32-zlib) [ 3252.271762] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3252.522831] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3254.224288] Key type lgssc registered [ 3255.368271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3263.286188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3263.306597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.775893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.173812] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3274.215063] LustreError: 86342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.227966] LustreError: 86342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3275.339158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3275.715646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3279.616127] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3284.742316] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3288.483752] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3293.631653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3293.912976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.972127] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3304.098498] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3309.302229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3319.783138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.787611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3319.809096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.901479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.912282] Lustre: Skipped 1 previous similar message [ 3330.026124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.034270] Lustre: Skipped 1 previous similar message [ 3333.088196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3333.136241] LustreError: 88606:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3333.238391] Lustre: server umount lustre-OST0000 complete [ 3337.185873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3337.189556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.196795] Lustre: Skipped 1 previous similar message [ 3337.202467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3337.208149] Lustre: Skipped 1 previous similar message [ 3340.272386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.692234] LustreError: 88808:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3342.695192] LustreError: 88808:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3342.753551] Lustre: server umount lustre-MDT0000 complete [ 3345.379909] LustreError: 86341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.396233] LustreError: 86341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3345.880109] LustreError: 86321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687638 with bad export cookie 4340036219856407375 [ 3345.886096] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3345.887488] LustreError: 86321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3346.014305] LustreError: 89009:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3346.017025] LustreError: 89009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3346.124908] Lustre: server umount lustre-MDT0001 complete [ 3354.270554] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3356.910764] Key type lgssc unregistered [ 3357.173350] LNet: 89544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3357.189839] LNetError: 89544:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3357.209796] LNet: Removed LNI 192.168.201.128@tcp [ 3357.972253] Key type .llcrypt unregistered [ 3357.974359] Key type ._llcrypt unregistered [ 3374.900317] Key type ._llcrypt registered [ 3374.903150] Key type .llcrypt registered [ 3374.982886] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3375.674565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3375.693877] alg: No test for adler32 (adler32-zlib) [ 3376.690777] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3376.897075] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3378.552653] Key type lgssc registered [ 3379.363962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3385.408996] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3385.440239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3400.993060] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3406.112542] LustreError: 90890:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 3432.800193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.999786] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3442.652378] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.350509] LustreError: 91745:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3447.413914] Lustre: server umount lustre-OST0000 complete [ 3454.155230] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:42:24 (1770687744) [ 3459.376268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.687498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.799992] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3465.196268] LustreError: 92413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.217890] LustreError: 92413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3469.285232] LustreError: 92413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.657322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3469.915887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.990141] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3477.296864] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3480.608685] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.511325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.810613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.109680] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3496.819903] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3504.240859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3514.852721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.854376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.858734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3514.865217] Lustre: Skipped 1 previous similar message [ 3514.887825] Lustre: Skipped 1 previous similar message [ 3519.982540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3519.991308] Lustre: Skipped 1 previous similar message [ 3520.567250] LustreError: 94837:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3520.571477] LustreError: 94837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3520.660678] Lustre: server umount lustre-OST0000 complete [ 3525.090356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.094457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3525.106504] Lustre: Skipped 1 previous similar message [ 3530.082309] LustreError: 95038:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3530.085616] LustreError: 95038:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3530.182350] Lustre: server umount lustre-MDT0000 complete [ 3530.209253] LustreError: 92409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.232781] LustreError: 92409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3533.203777] LustreError: 92393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687826 with bad export cookie 5260397045204348597 [ 3533.213834] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3533.216601] LustreError: 92393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3533.375459] LustreError: 95239:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3533.387464] LustreError: 95239:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3533.564554] Lustre: server umount lustre-MDT0001 complete [ 3541.960025] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3544.291125] Key type lgssc unregistered [ 3544.599376] LNet: 95772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3544.611556] LNetError: 95772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3544.647561] LNet: Removed LNI 192.168.201.128@tcp [ 3545.344990] Key type .llcrypt unregistered [ 3545.347114] Key type ._llcrypt unregistered [ 3565.142552] Key type ._llcrypt registered [ 3565.144526] Key type .llcrypt registered [ 3565.253560] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3566.302653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3566.349922] alg: No test for adler32 (adler32-zlib) [ 3567.415211] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3567.642266] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3569.312178] Key type lgssc registered [ 3570.375910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3577.030203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3577.054587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.436543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.911437] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3588.580900] LustreError: 97146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.594082] LustreError: 97146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3589.096884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3589.369370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3592.281772] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3595.885432] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3598.671956] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3604.116847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3604.454343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.337926] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3613.846471] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3623.930544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3634.152941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3634.153615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.161864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3634.178960] Lustre: Skipped 1 previous similar message [ 3636.866070] LustreError: 99406:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3636.993608] Lustre: server umount lustre-OST0000 complete [ 3640.802911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3640.810202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.825109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.829393] Lustre: Skipped 1 previous similar message [ 3644.387854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3644.388225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.508757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.519221] Lustre: Skipped 2 previous similar messages [ 3654.114420] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3654.194848] LustreError: 99609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3654.200176] LustreError: 99609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3654.251971] Lustre: server umount lustre-MDT0000 complete [ 3654.629994] LustreError: 98253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.645336] LustreError: 98253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3656.641887] LustreError: 99610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687949 with bad export cookie 16849186536599875993 [ 3656.646589] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.652992] LustreError: 99610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3656.775882] LustreError: 99813:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3656.782706] LustreError: 99813:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3656.887960] Lustre: server umount lustre-MDT0001 complete [ 3664.044493] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3666.221070] Key type lgssc unregistered [ 3666.490539] LNet: 100344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3666.501541] LNetError: 100344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.515904] LNet: Removed LNI 192.168.201.128@tcp [ 3667.130791] Key type .llcrypt unregistered [ 3667.133094] Key type ._llcrypt unregistered [ 3680.914698] Key type ._llcrypt registered [ 3680.917411] Key type .llcrypt registered [ 3681.025158] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3681.724778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3681.742621] alg: No test for adler32 (adler32-zlib) [ 3682.765129] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3682.934930] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3684.545737] Key type lgssc registered [ 3685.416191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3691.282792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3691.305365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3706.848923] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3711.968254] LustreError: 101546:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 3738.592679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.445666] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3746.583912] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3751.306182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.488318] LustreError: MGC192.168.201.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3751.501179] LustreError: 102455:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3751.510497] LustreError: 102455:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 3751.515960] LustreError: 102455:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3751.566544] Lustre: server umount lustre-MDT0000 complete [ 3751.568232] LustreError: 102455:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3766.243044] LustreError: 102885:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3766.311364] Lustre: server umount lustre-OST0000 complete [ 3772.940962] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 3780.421823] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3806.999722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.139662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3807.164202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3807.250872] Lustre: lustre-MDT0000: new disk, initializing [ 3807.327226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.351927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3810.529226] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3818.025059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3818.143884] Lustre: 105218:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3818.206000] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3818.210186] Lustre: Skipped 1 previous similar message [ 3818.276389] Lustre: lustre-MDT0001: new disk, initializing [ 3818.340691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3818.370606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3818.382581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3821.177734] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3826.970093] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3829.472720] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3834.446044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3834.662232] Lustre: lustre-OST0000: new disk, initializing [ 3834.665118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3834.673884] Lustre: 106960:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3834.722466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.059093] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3841.576486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3841.583394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3841.655095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3845.075900] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3848.353682] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3848.513600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3851.667119] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3851.842899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3856.874182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.876923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3856.893092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.903502] Lustre: Skipped 1 previous similar message [ 3859.456165] LustreError: 108150:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3859.461599] LustreError: 108150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3859.599161] Lustre: server umount lustre-OST0000 complete [ 3864.545912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3864.552653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.561158] Lustre: Skipped 1 previous similar message [ 3864.566270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3867.110840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3867.111684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.443235] LustreError: 108352:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3868.448427] LustreError: 108352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3868.515967] Lustre: server umount lustre-MDT0000 complete [ 3870.855495] LustreError: 105994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688163 with bad export cookie 3199834436232807782 [ 3870.861304] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3870.865605] LustreError: 105994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3870.946967] LustreError: 108554:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3870.949952] LustreError: 108554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3871.033153] Lustre: server umount lustre-MDT0001 complete [ 3876.757789] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:49:27 (1770688167) [ 3879.054556] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3889.110972] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 3895.092143] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3916.610367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.782762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3916.812695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3916.867698] Lustre: lustre-MDT0000: new disk, initializing [ 3916.920146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.937636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3919.234446] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3925.262524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3925.330976] Lustre: 111673:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3925.378539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3925.382205] Lustre: Skipped 1 previous similar message [ 3925.432476] Lustre: lustre-MDT0001: new disk, initializing [ 3925.483160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.502906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3925.509158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3927.382601] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3931.554473] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3933.383327] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3936.596528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3936.746635] Lustre: lustre-OST0000: new disk, initializing [ 3936.750089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3936.755713] Lustre: 113412:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.665767] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3942.410770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3942.418935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3942.487211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3944.067883] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3946.396689] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3946.509940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3949.047445] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3949.200806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3952.612541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3952.619119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.624190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.634799] Lustre: Skipped 1 previous similar message [ 3952.634946] Lustre: Skipped 1 previous similar message [ 3956.413581] LustreError: 114602:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3956.417609] LustreError: 114602:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3956.491294] Lustre: server umount lustre-OST0000 complete [ 3961.313299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3961.317801] LustreError: Skipped 1 previous similar message [ 3961.320432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.331645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.334164] Lustre: Skipped 1 previous similar message [ 3972.069638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3972.074136] Lustre: Skipped 5 previous similar messages [ 3973.089587] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3973.135664] LustreError: 114803:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3973.139354] LustreError: 114803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3973.181512] Lustre: server umount lustre-MDT0000 complete [ 3975.088660] LustreError: 112451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688268 with bad export cookie 3199834436232808664 [ 3975.094592] LustreError: 112451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3975.094969] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.295190] Lustre: server umount lustre-MDT0001 complete [ 3980.127578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.563631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.568700] Lustre: Skipped 1 previous similar message [ 3982.902342] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3985.889743] LustreError: 115479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.899354] LustreError: 115479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3987.661418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3989.957989] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3993.496593] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3995.787630] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4000.385956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4004.438306] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4007.974243] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4016.356138] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4018.150118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4018.150394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.176508] Lustre: Skipped 1 previous similar message [ 4018.181359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.080433] LustreError: 117816:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4020.085432] LustreError: 117816:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4020.151097] Lustre: server umount lustre-OST0000 complete [ 4028.541180] Lustre: server umount lustre-MDT0000 complete [ 4030.318810] LustreError: 115460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688323 with bad export cookie 3199834436232809546 [ 4030.321811] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4030.324335] LustreError: 115460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4030.493560] Lustre: server umount lustre-MDT0001 complete [ 4036.946645] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4038.562639] Key type lgssc unregistered [ 4038.764049] LNet: 118751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4038.769639] LNetError: 118751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4038.783789] LNet: Removed LNI 192.168.201.128@tcp [ 4039.248919] Key type .llcrypt unregistered [ 4039.250320] Key type ._llcrypt unregistered [ 4049.028689] Key type ._llcrypt registered [ 4049.030963] Key type .llcrypt registered [ 4049.094556] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 4057.913908] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4058.373725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4058.444749] alg: No test for adler32 (adler32-zlib) [ 4059.361355] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4059.496798] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4061.088166] Key type lgssc registered [ 4061.673604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4081.504488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4081.515548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.701343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4082.728793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4082.800471] Lustre: lustre-MDT0000: new disk, initializing [ 4082.869056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.880080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4085.019318] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4091.342695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4091.426817] Lustre: 121842:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4091.462140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4091.465091] Lustre: Skipped 1 previous similar message [ 4091.510909] Lustre: lustre-MDT0001: new disk, initializing [ 4091.569169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.591364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4091.597282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4093.648395] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4098.246843] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4100.371353] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4104.496464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4104.690565] Lustre: lustre-OST0000: new disk, initializing [ 4104.694170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4104.699663] Lustre: 123582:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4104.749914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4108.164274] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4111.922876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4111.931766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4111.996635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4112.948308] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4115.248058] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4115.382677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4117.700204] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4117.841335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4122.088358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.089034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4122.097237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4122.105290] Lustre: Skipped 1 previous similar message [ 4127.207057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.215867] Lustre: Skipped 1 previous similar message [ 4132.323596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4132.326608] Lustre: Skipped 1 previous similar message [ 4133.344159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4133.377589] LustreError: 124771:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4133.446716] Lustre: server umount lustre-OST0000 complete [ 4137.442491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.454880] Lustre: Skipped 1 previous similar message [ 4137.460026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.465249] Lustre: Skipped 1 previous similar message [ 4141.782071] LustreError: 124976:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4141.786231] LustreError: 124976:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4141.878573] Lustre: server umount lustre-MDT0000 complete [ 4144.140447] LustreError: 121835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688437 with bad export cookie 735208412869007167 [ 4144.143266] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4144.150286] LustreError: 121835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4144.306484] LustreError: 125177:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4144.316093] LustreError: 125177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4144.447977] Lustre: server umount lustre-MDT0001 complete [ 4149.495084] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:54:00 (1770688440) [ 4153.849266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.200645] LustreError: 125847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.210826] LustreError: 125847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4154.269174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.523137] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4159.459436] LustreError: 125847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.886974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4162.122810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4164.641234] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4167.593087] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4169.488809] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4172.258563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4172.263188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.273334] Lustre: Skipped 1 previous similar message [ 4172.277812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4176.782854] LustreError: 127187:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4176.785245] LustreError: 127187:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4176.841629] Lustre: server umount lustre-MDT0000 complete [ 4178.401915] LustreError: 125846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.411125] LustreError: 125846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4178.775234] LustreError: 125827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688471 with bad export cookie 735208412869008049 [ 4178.779665] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.782730] LustreError: 125827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4178.937671] Lustre: server umount lustre-MDT0001 complete [ 4183.542886] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:54:34 (1770688474) [ 4187.386783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4202.912960] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4208.032690] LustreError: 128034:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 4233.696794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.733774] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4239.457718] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4240.415538] LustreError: 128695:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4240.418981] LustreError: 128695:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4240.470525] Lustre: server umount lustre-OST0000 complete [ 4244.285142] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:55:35 (1770688535) [ 4247.348692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.579207] LustreError: 129366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.615618] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4252.641391] LustreError: 129366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.871871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4254.080424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4254.085527] Lustre: Skipped 1 previous similar message [ 4256.253784] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4259.320477] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4261.508812] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4265.503864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4269.365261] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4273.253841] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.395758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.399740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.409464] Lustre: Skipped 3 previous similar messages [ 4284.413057] Lustre: Skipped 3 previous similar messages [ 4289.616049] LustreError: 131607:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4289.620818] LustreError: 131607:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4289.668220] Lustre: server umount lustre-MDT0000 complete [ 4291.956713] LustreError: 129347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688584 with bad export cookie 735208412869008714 [ 4291.962755] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.964278] LustreError: 129347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4292.158728] Lustre: server umount lustre-MDT0001 complete [ 4304.837039] Lustre: server umount lustre-OST0000 complete [ 4309.236777] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:56:40 (1770688600) [ 4312.475871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.732591] LustreError: 132682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.765661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.770299] Lustre: Skipped 1 previous similar message [ 4314.776697] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4319.294905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4321.720869] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4324.718519] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4326.968994] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4330.664644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4333.715745] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4336.858860] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4338.027340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4338.982032] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4339.093318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4341.319972] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4344.514235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 4348.386656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4348.390901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.397785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4348.400938] Lustre: Skipped 4 previous similar messages [ 4351.539093] LustreError: 135440:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4351.542894] LustreError: 135440:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4351.600613] Lustre: server umount lustre-OST0000 complete [ 4359.650504] LustreError: 132700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.659504] LustreError: 132700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4359.672570] Lustre: server umount lustre-MDT0000 complete [ 4361.587712] LustreError: 132661:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688654 with bad export cookie 735208412869009764 [ 4361.591684] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4361.594720] LustreError: 132661:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4361.768700] Lustre: server umount lustre-MDT0001 complete [ 4366.207982] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:57:37 (1770688657) [ 4369.768534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4385.316336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4390.432793] LustreError: 136486:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 4416.096534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.100987] Lustre: Skipped 2 previous similar messages [ 4419.060434] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4421.953249] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4425.865284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.134407] LustreError: 136512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4426.232979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4428.393055] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4433.354220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4435.953449] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4439.111345] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4441.274714] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4443.606931] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4443.753610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4446.200326] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4446.323076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4449.254529] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.260411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4449.270064] Lustre: Skipped 3 previous similar messages [ 4449.273523] LustreError: Skipped 1 previous similar message [ 4449.277816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.283361] Lustre: Skipped 6 previous similar messages [ 4453.524156] LustreError: 139254:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4453.529056] LustreError: 139254:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4453.588973] Lustre: server umount lustre-MDT0000 complete [ 4455.930184] LustreError: 139257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688748 with bad export cookie 735208412869010779 [ 4455.942538] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.141562] Lustre: server umount lustre-MDT0001 complete [ 4468.390747] Lustre: server umount lustre-OST0000 complete [ 4472.638749] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:59:23 (1770688763) [ 4476.483720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4492.000761] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4497.120387] LustreError: 140303:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 4525.399216] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4528.109432] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4531.431668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.664533] LustreError: 140329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.674532] LustreError: 140329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4531.746732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4533.612263] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4537.991336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4540.313910] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4543.317476] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4545.111609] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4548.587273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4548.691736] Lustre: lustre-OST0001: new disk, initializing [ 4548.695883] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4548.700984] Lustre: 142836:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4548.763317] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4548.772097] Lustre: Skipped 5 previous similar messages [ 4551.826312] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4555.788030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4555.793460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4555.795311] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4555.862341] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4557.762189] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4557.872866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4559.896220] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4559.998201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4562.402478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4562.406321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.411160] Lustre: Skipped 2 previous similar messages [ 4562.413975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4575.200407] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4575.219103] LustreError: 144027:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4575.222952] LustreError: 144027:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4575.246760] Lustre: server umount lustre-OST0000 complete [ 4581.346762] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.353309] LustreError: Skipped 2 previous similar messages [ 4583.262561] Lustre: server umount lustre-OST0001 complete [ 4599.264385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.354122] Lustre: server umount lustre-MDT0000 complete [ 4601.340935] LustreError: 140309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688894 with bad export cookie 735208412869011724 [ 4601.343316] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.349796] LustreError: 140309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4601.517272] Lustre: server umount lustre-MDT0001 complete [ 4612.460376] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:01:43 (1770688903) [ 4613.266682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4614.057719] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4614.894228] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:01:45 (1770688905) [ 4618.019996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.158169] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4618.174787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4620.089335] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4623.805953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4623.847434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4623.859898] Lustre: 146257:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4625.745577] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4628.022190] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4629.622291] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4632.457491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4632.551911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4634.328572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4635.028102] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4637.349911] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4639.208354] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4639.302842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4641.061857] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4641.176610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4644.835500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4644.836023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.839749] LustreError: Skipped 1 previous similar message [ 4644.845085] Lustre: Skipped 5 previous similar messages [ 4648.155697] Lustre: server umount lustre-OST0000 complete [ 4662.868409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4664.102614] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4664.108275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4664.108340] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4664.117836] Lustre: Skipped 1 previous similar message [ 4665.591984] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4667.831931] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4669.571057] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4669.653081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4671.333258] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4671.419618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4672.997524] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95fc05bf6000.ost_server_uuid 50 [ 4673.699255] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95fc05bf6000.ost_server_uuid in FULL state after 0 sec [ 4673.767793] LustreError: 146265:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.28@tcp) returned error from blocking AST (req@00000000a4f1f471 x1856701309595904 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b1445258/0xa33fc27fcefb251 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.28@tcp remote: 0x960513b603e4e061 expref: 5 pid: 146265 timeout: 4773 lvb_type: 0 [ 4673.783032] LustreError: lustre-MDT0000: A client on nid 192.168.201.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4673.786723] LustreError: 146253:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.28@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b1445258/0xa33fc27fcefb251 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.28@tcp remote: 0x960513b603e4e061 expref: 6 pid: 146265 timeout: 0 lvb_type: 0 [ 4679.655843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4679.660940] LustreError: Skipped 1 previous similar message [ 4681.317705] Lustre: server umount lustre-OST0000 complete [ 4689.198398] Lustre: server umount lustre-MDT0000 complete [ 4689.890729] LustreError: 147108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4689.898209] LustreError: 147108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4690.813117] LustreError: 146250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688983 with bad export cookie 735208412869012900 [ 4690.816954] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.819991] LustreError: 146250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4705.248326] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4705.292032] LustreError: 151201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4705.294754] LustreError: 151201:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 4705.358668] Lustre: server umount lustre-MDT0001 complete [ 4709.716784] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4710.984740] Key type lgssc unregistered [ 4711.126635] LNet: 151734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4711.128984] LNetError: 151734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4711.138501] LNet: Removed LNI 192.168.201.128@tcp [ 4711.506369] Key type .llcrypt unregistered [ 4711.508821] Key type ._llcrypt unregistered [ 4722.050993] Key type ._llcrypt registered [ 4722.052762] Key type .llcrypt registered [ 4722.105518] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4722.747971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4722.758423] alg: No test for adler32 (adler32-zlib) [ 4723.651216] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4723.752043] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4725.344345] Key type lgssc registered [ 4725.813266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4729.009250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4729.020495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.220777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.933974] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4735.640770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4735.807490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4737.524722] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4739.978626] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4741.605564] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4744.452303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4744.599023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.083467] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4749.629747] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4755.886578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4758.503539] Lustre: Failing over lustre-MDT0000 [ 4758.733960] LustreError: 155346:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4758.770380] Lustre: server umount lustre-MDT0000 complete [ 4761.059720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.060971] LustreError: 153127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.066482] Lustre: Skipped 2 previous similar messages [ 4761.076817] LustreError: 153127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4766.178181] LustreError: 154309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.183638] LustreError: 154309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4769.828777] LustreError: 153084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.28@tcp arrived at 1770689062 with bad export cookie 6678828105694983124 [ 4769.835537] LustreError: 153084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4771.297520] LustreError: 154309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.303791] LustreError: 154309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4772.706404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.753909] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4772.873398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.905385] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4774.580675] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4775.488760] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4775.492962] Lustre: lustre-MDT0000: Denying connection for new client 60e2ae2a-3a01-44b4-a26a-56e9550a327e (at 192.168.201.28@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 4777.964118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4783.074939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.082914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.085405] Lustre: Skipped 1 previous similar message [ 4788.194556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4788.198088] Lustre: Skipped 1 previous similar message [ 4788.464123] LustreError: 156280:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4788.466806] LustreError: 156280:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4788.512869] Lustre: server umount lustre-OST0000 complete [ 4789.939094] LustreError: 156481:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4789.941554] Lustre: 155848:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4789.945752] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4789.957232] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4789.963821] LustreError: 155848:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000073c63f2 x1856702005676544/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4789.971720] LustreError: 155848:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4789.975480] LustreError: 155848:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4789.979261] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 4790.071711] LustreError: 156481:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4790.075383] LustreError: 156481:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4790.110674] Lustre: server umount lustre-MDT0000 complete [ 4791.529465] LustreError: 153082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689084 with bad export cookie 6678828105694983481 [ 4791.534366] LustreError: 152414:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4791.535358] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.537176] LustreError: 153082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4797.634678] LustreError: 156682:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4797.637873] LustreError: 156682:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4797.706267] Lustre: server umount lustre-MDT0001 complete [ 4802.031790] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4803.316184] Key type lgssc unregistered [ 4803.466643] LNet: 157214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4803.469296] LNetError: 157214:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4803.482600] LNet: Removed LNI 192.168.201.128@tcp [ 4803.834907] Key type .llcrypt unregistered [ 4803.836891] Key type ._llcrypt unregistered [ 4814.482040] Key type ._llcrypt registered [ 4814.484037] Key type .llcrypt registered [ 4814.533437] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4814.952446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4814.962586] alg: No test for adler32 (adler32-zlib) [ 4815.838890] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4815.951582] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4817.544176] Key type lgssc registered [ 4818.012351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4821.124987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4821.134588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.323337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.019835] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4827.551722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4827.718277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4829.295613] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4831.590538] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4833.180634] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4835.999414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4836.202032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4838.763805] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4840.948697] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4848.489915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4853.731898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.734133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4853.737842] Lustre: Skipped 1 previous similar message [ 4853.743428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4854.960329] LustreError: 160848:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4855.000403] Lustre: server umount lustre-OST0000 complete [ 4858.336904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4858.340387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.348155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4858.350971] Lustre: Skipped 1 previous similar message [ 4858.495814] LustreError: 161051:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4858.498069] LustreError: 161051:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4858.523917] Lustre: server umount lustre-MDT0000 complete [ 4858.851092] LustreError: 158580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.857304] LustreError: 158580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4859.789989] LustreError: 159302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689152 with bad export cookie 5528383402306119386 [ 4859.793183] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4859.794675] LustreError: 159302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4859.868200] LustreError: 161253:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4859.870859] LustreError: 161253:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4859.927703] Lustre: server umount lustre-MDT0001 complete [ 4863.941045] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4865.072209] Key type lgssc unregistered [ 4865.210370] LNet: 161783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4865.214140] LNetError: 161783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4865.221436] LNet: Removed LNI 192.168.201.128@tcp [ 4865.550781] Key type .llcrypt unregistered [ 4865.551863] Key type ._llcrypt unregistered [ 4878.936928] Key type ._llcrypt registered [ 4878.938049] Key type .llcrypt registered [ 4878.978420] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4879.350736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4879.403086] alg: No test for adler32 (adler32-zlib) [ 4880.268654] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4880.358208] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4881.944214] Key type lgssc registered [ 4882.364272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4885.485800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4885.494841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.692141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4888.257777] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4891.723797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4891.885919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.432668] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4895.778767] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4897.552807] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4900.531892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4900.707290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4903.208252] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4905.863229] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.984194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4922.338945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4922.339978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.342891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.343356] LustreError: Skipped 1 previous similar message [ 4922.348495] Lustre: Skipped 1 previous similar message [ 4924.208109] LustreError: 165704:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4924.264497] Lustre: server umount lustre-OST0000 complete [ 4927.458366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.465690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.468920] Lustre: Skipped 2 previous similar messages [ 4932.578213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.581594] Lustre: Skipped 1 previous similar message [ 4937.697565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4937.701558] Lustre: Skipped 1 previous similar message [ 4940.256128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4940.294933] LustreError: 165906:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4940.297035] LustreError: 165906:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4940.326758] Lustre: server umount lustre-MDT0000 complete [ 4941.895614] LustreError: 163420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689234 with bad export cookie 6792772924226053115 [ 4941.898837] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.900817] LustreError: 163420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4941.979958] LustreError: 166109:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4941.982953] LustreError: 166109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4942.042119] Lustre: server umount lustre-MDT0001 complete [ 4946.404943] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4947.652076] Key type lgssc unregistered [ 4947.800382] LNet: 166640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4947.804122] LNetError: 166640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4947.811368] LNet: Removed LNI 192.168.201.128@tcp [ 4948.140479] Key type .llcrypt unregistered [ 4948.142209] Key type ._llcrypt unregistered [ 4958.263191] Key type ._llcrypt registered [ 4958.265184] Key type .llcrypt registered [ 4958.308203] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4958.761992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4958.789076] alg: No test for adler32 (adler32-zlib) [ 4959.673786] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4959.780303] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4961.376118] Key type lgssc registered [ 4961.868623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4966.540961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4966.552993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4967.777752] Lustre: *** cfs_fail_loc=135, val=0*** [ 4967.828837] LustreError: 168121:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4967.832224] LustreError: 168121:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.128@tcp: cfg command failed: rc = -2 [ 4967.836943] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4967.836943] [ 4967.840751] LustreError: MGC192.168.201.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4967.848220] LustreError: 168081:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4967.851315] LustreError: 168081:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 4967.854343] LustreError: 168081:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 4967.856654] LustreError: 168081:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 4967.859850] LustreError: 168081:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4967.905777] Lustre: server umount lustre-MDT0000 complete [ 4967.907745] LustreError: 168081:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4970.846657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.070835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.678805] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4976.140190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4976.296628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4977.888959] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4979.819395] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4981.167841] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4985.827210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4985.830432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4985.834266] Lustre: Skipped 1 previous similar message [ 4985.840467] Lustre: Skipped 1 previous similar message [ 4986.850819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.853163] Lustre: Skipped 1 previous similar message [ 4988.036929] LustreError: 169831:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4988.040206] LustreError: 169831:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4988.076304] Lustre: server umount lustre-MDT0000 complete [ 4989.482066] LustreError: 169833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689282 with bad export cookie 1978196303019241347 [ 4989.485468] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.487257] LustreError: 169833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4989.565676] LustreError: 170036:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4989.568763] LustreError: 170036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4989.629666] Lustre: server umount lustre-MDT0001 complete [ 4994.150639] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4995.516661] Key type lgssc unregistered [ 4995.666394] LNet: 170567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4995.670298] LNetError: 170567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4995.684554] LNet: Removed LNI 192.168.201.128@tcp [ 4996.049878] Key type .llcrypt unregistered [ 4996.051526] Key type ._llcrypt unregistered [ 5004.799102] Key type ._llcrypt registered [ 5004.800474] Key type .llcrypt registered [ 5004.848853] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5004.974404] Key type .llcrypt unregistered [ 5004.975981] Key type ._llcrypt unregistered [ 5012.961747] Key type ._llcrypt registered [ 5012.963457] Key type .llcrypt registered [ 5013.011121] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5013.470848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5013.513202] alg: No test for adler32 (adler32-zlib) [ 5014.369381] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5014.461205] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5016.056155] Key type lgssc registered [ 5016.535862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5019.790155] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5019.797990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5035.232545] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5040.352746] LustreError: 172634:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 5066.976166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.331365] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5071.822948] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5074.716443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.928196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5074.961138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5076.658263] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5080.039293] LustreError: 172660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.047148] LustreError: 172660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5080.356542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5080.548611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5082.313212] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5084.682917] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5086.315796] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5087.478694] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5090.272974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.277033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.283598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.785790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.790248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.795469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.810521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.813911] Lustre: Skipped 1 previous similar message [ 5096.046128] LustreError: 175158:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5096.087913] Lustre: server umount lustre-OST0000 complete [ 5101.025258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5101.029451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.036881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.631091] LustreError: 175360:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5103.636120] LustreError: 175360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5103.680522] Lustre: server umount lustre-MDT0000 complete [ 5105.292898] LustreError: 172640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689398 with bad export cookie 6167623664972807386 [ 5105.295407] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.300464] LustreError: 172640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5105.387350] LustreError: 175563:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5105.393215] LustreError: 175563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5105.472027] Lustre: server umount lustre-MDT0001 complete [ 5110.152821] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5111.316669] Key type lgssc unregistered [ 5111.474456] LNet: 176094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5111.479408] LNetError: 176094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5111.490448] LNet: Removed LNI 192.168.201.128@tcp [ 5111.863678] Key type .llcrypt unregistered [ 5111.866050] Key type ._llcrypt unregistered [ 5122.153570] Key type ._llcrypt registered [ 5122.155447] Key type .llcrypt registered [ 5122.199751] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5122.615733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5122.661040] alg: No test for adler32 (adler32-zlib) [ 5123.535322] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5123.637455] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5125.232137] Key type lgssc registered [ 5125.742955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5128.929195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5128.935986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.122205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.848531] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5135.743990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5135.927890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5137.663568] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5139.976576] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5141.479532] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5144.218049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5144.353756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.582509] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5148.575810] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5156.651716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5160.569081] Lustre: Failing over lustre-MDT0000 [ 5160.655332] LustreError: 179777:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5160.689608] Lustre: server umount lustre-MDT0000 complete [ 5161.441045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5161.444113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.449689] LustreError: 178667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.457103] LustreError: 178667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5161.954859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.955933] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.960930] Lustre: Skipped 1 previous similar message [ 5164.259167] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.268158] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5167.074320] LustreError: 178667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.081634] LustreError: 178667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5172.193841] LustreError: 178667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.202250] LustreError: 178667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5174.143582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5174.197745] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5174.301739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5174.333427] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5174.491188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5176.153215] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5178.251223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5179.368026] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5179.376091] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5179.402424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5180.481042] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5189.602980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5189.604287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.608041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5189.613348] Lustre: Skipped 1 previous similar message [ 5194.722997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.726852] Lustre: Skipped 2 previous similar messages [ 5195.317133] LustreError: 180962:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5195.320954] LustreError: 180962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5195.374771] Lustre: server umount lustre-OST0000 complete [ 5199.844107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.844857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5199.848312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5199.858568] Lustre: Skipped 1 previous similar message [ 5203.266721] LustreError: 181163:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5203.269947] LustreError: 181163:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5203.300171] Lustre: server umount lustre-MDT0000 complete [ 5204.845644] LustreError: 180856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689497 with bad export cookie 2652347174015986169 [ 5204.848740] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.850976] LustreError: 180856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5204.990795] Lustre: server umount lustre-MDT0001 complete [ 5209.846830] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5211.073153] Key type lgssc unregistered [ 5211.227579] LNet: 181896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5211.232114] LNetError: 181896:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5211.242301] LNet: Removed LNI 192.168.201.128@tcp [ 5211.591337] Key type .llcrypt unregistered [ 5211.593666] Key type ._llcrypt unregistered [ 5222.141891] Key type ._llcrypt registered [ 5222.143204] Key type .llcrypt registered [ 5222.191252] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5222.556198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5222.596196] alg: No test for adler32 (adler32-zlib) [ 5223.459074] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5223.549914] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5225.136119] Key type lgssc registered [ 5225.580085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5228.623918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5228.633188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.819695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.231143] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5234.463088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5234.623486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.218594] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5238.564499] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5240.218258] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5242.934724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5243.062621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.271252] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5247.361464] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5254.312761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5258.773100] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5262.303660] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5270.873146] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5280.229301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.229350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5280.229903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5280.235695] Lustre: Skipped 1 previous similar message [ 5280.239927] LustreError: Skipped 1 previous similar message [ 5284.594994] LustreError: 185706:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5284.635634] Lustre: server umount lustre-OST0000 complete [ 5290.466780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.468728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5290.475406] Lustre: Skipped 1 previous similar message [ 5290.481471] Lustre: Skipped 2 previous similar messages [ 5292.299668] LustreError: 185907:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5292.303173] LustreError: 185907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5292.330695] Lustre: server umount lustre-MDT0000 complete [ 5294.023788] LustreError: 183989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689586 with bad export cookie 9245585293944613556 [ 5294.026341] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.031289] LustreError: 183989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5294.116840] LustreError: 186109:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5294.120312] LustreError: 186109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5294.188414] Lustre: server umount lustre-MDT0001 complete [ 5299.033179] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5300.429859] Key type lgssc unregistered [ 5300.593547] LNet: 186640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5300.597347] LNetError: 186640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5300.609230] LNet: Removed LNI 192.168.201.128@tcp [ 5301.074212] Key type .llcrypt unregistered [ 5301.075973] Key type ._llcrypt unregistered [ 5312.856823] Key type ._llcrypt registered [ 5312.858747] Key type .llcrypt registered [ 5312.917175] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5313.343967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5313.352026] alg: No test for adler32 (adler32-zlib) [ 5314.248189] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5314.353424] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5315.952638] Key type lgssc registered [ 5316.519981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5320.370792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5320.380966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.634327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5323.573292] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5327.725256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5327.911464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5329.598217] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5331.975853] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5333.688307] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5336.762226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5336.934054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.438242] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5341.879361] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.155135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5349.309411] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5351.075522] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5360.265411] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5364.823411] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5371.875402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5371.876073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.877038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5371.890663] Lustre: Skipped 1 previous similar message [ 5376.995670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5376.998089] Lustre: Skipped 1 previous similar message [ 5377.019789] LustreError: 191659:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5377.066119] Lustre: server umount lustre-OST0000 complete [ 5379.041132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5379.044748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.052218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5382.113867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5382.115315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.233791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.236358] Lustre: Skipped 2 previous similar messages [ 5393.376121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5393.426564] LustreError: 191861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5393.429512] LustreError: 191861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5393.461285] Lustre: server umount lustre-MDT0000 complete [ 5395.205972] LustreError: 188004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689688 with bad export cookie 4690605479542798676 [ 5395.207515] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.214211] LustreError: 188004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5395.293861] LustreError: 192064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5395.296365] LustreError: 192064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5395.368268] Lustre: server umount lustre-MDT0001 complete [ 5400.159460] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5401.573345] Key type lgssc unregistered [ 5401.744958] LNet: 192595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5401.749215] LNetError: 192595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5401.762397] LNet: Removed LNI 192.168.201.128@tcp [ 5402.191064] Key type .llcrypt unregistered [ 5402.192981] Key type ._llcrypt unregistered [ 5413.190738] Key type ._llcrypt registered [ 5413.191881] Key type .llcrypt registered [ 5413.239403] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5413.708255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5413.726673] alg: No test for adler32 (adler32-zlib) [ 5414.605097] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5414.702176] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5416.296358] Key type lgssc registered [ 5416.797094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5420.405091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5420.415203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.648170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.422697] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5427.131082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5427.303053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5429.113110] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5431.635919] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5433.178203] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5436.202184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5436.354308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.716019] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5440.994627] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5446.638389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5456.869299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.871115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5456.872342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5456.876762] Lustre: Skipped 1 previous similar message [ 5458.031391] LustreError: 196379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5458.083454] Lustre: server umount lustre-OST0000 complete [ 5461.987218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.990692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5461.994227] Lustre: Skipped 1 previous similar message [ 5462.000339] Lustre: Skipped 2 previous similar messages [ 5462.087392] LustreError: 196579:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5462.090368] LustreError: 196579:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5462.132779] Lustre: server umount lustre-MDT0000 complete [ 5463.888813] LustreError: 193946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689756 with bad export cookie 8462131375699886534 [ 5463.894142] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.900792] LustreError: 193946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.983794] LustreError: 196782:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5463.987300] LustreError: 196782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5464.106117] Lustre: server umount lustre-MDT0001 complete [ 5469.122413] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5470.503602] Key type lgssc unregistered [ 5470.660371] LNet: 197314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.665315] LNetError: 197314:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.679432] LNet: Removed LNI 192.168.201.128@tcp [ 5471.066518] Key type .llcrypt unregistered [ 5471.068547] Key type ._llcrypt unregistered [ 5479.893611] Key type ._llcrypt registered [ 5479.895281] Key type .llcrypt registered [ 5479.938176] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5480.342654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.402188] alg: No test for adler32 (adler32-zlib) [ 5481.275943] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5481.374151] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5482.960160] Key type lgssc registered [ 5483.361937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.213944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.222186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.430521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.128094] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5492.435189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5492.607715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5494.281348] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5496.368426] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5497.813361] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5500.496828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5500.642226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.835204] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5505.144791] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5512.935874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5523.429302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5523.429501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.430459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5523.437860] Lustre: Skipped 1 previous similar message [ 5526.641553] LustreError: 200945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5526.690340] Lustre: server umount lustre-OST0000 complete [ 5528.545875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.546274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5528.553053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5528.555784] LustreError: Skipped 1 previous similar message [ 5528.558628] Lustre: Skipped 2 previous similar messages [ 5533.666077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5533.669208] Lustre: Skipped 1 previous similar message [ 5534.408668] LustreError: 201147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5534.411316] LustreError: 201147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5534.447442] Lustre: server umount lustre-MDT0000 complete [ 5535.864210] LustreError: 200153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689828 with bad export cookie 4564054436158867840 [ 5535.868335] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5535.870246] LustreError: 200153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5535.947353] LustreError: 201349:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5535.951272] LustreError: 201349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5536.016226] Lustre: server umount lustre-MDT0001 complete [ 5540.344065] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5541.535834] Key type lgssc unregistered [ 5541.678694] LNet: 201880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.683293] LNetError: 201880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.695372] LNet: Removed LNI 192.168.201.128@tcp [ 5542.061550] Key type .llcrypt unregistered [ 5542.063284] Key type ._llcrypt unregistered [ 5550.426158] Key type ._llcrypt registered [ 5550.427919] Key type .llcrypt registered [ 5550.474165] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5550.849243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5550.894335] alg: No test for adler32 (adler32-zlib) [ 5551.777367] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5551.886591] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5553.480252] Key type lgssc registered [ 5553.974354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5557.193355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5557.202604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.405732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5560.095583] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5563.446834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5563.602144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.142657] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5567.262283] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.655335] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.944363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5571.073198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.193403] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5575.223307] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5583.395678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5588.510546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5588.549516] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5588.648403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5590.919780] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5593.100520] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5593.779057] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5604.449785] Lustre: Permanently deactivating lustre-OST0001 [ 5604.456852] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5604.902431] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5613.604117] Lustre: Permanently reactivating lustre-OST0001 [ 5613.612119] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5613.615103] Lustre: Skipped 2 previous similar messages [ 5620.731812] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.737575] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5620.742056] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5620.750408] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5620.765179] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5625.318863] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.322261] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5625.324015] Lustre: Skipped 2 previous similar messages [ 5625.329462] Lustre: Skipped 1 previous similar message [ 5629.230636] LustreError: 206562:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5629.253473] Lustre: server umount lustre-OST0001 complete [ 5630.437743] LustreError: 204698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5630.445884] LustreError: 204698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5634.529764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5634.533114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.538499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5634.540952] Lustre: Skipped 1 previous similar message [ 5635.556131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5635.556924] LustreError: 204697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.558546] Lustre: Skipped 1 previous similar message [ 5640.681784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5640.682823] LustreError: 204697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.684463] Lustre: Skipped 1 previous similar message [ 5640.692026] LustreError: 204697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5645.280165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.304079] LustreError: 206763:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5645.306434] LustreError: 206763:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5645.359317] Lustre: server umount lustre-OST0000 complete [ 5650.914396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5650.914559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.915396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5650.917752] LustreError: Skipped 1 previous similar message [ 5650.923502] Lustre: Skipped 2 previous similar messages [ 5661.152121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5661.153364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.158647] Lustre: Skipped 4 previous similar messages [ 5661.194377] LustreError: 206965:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5661.197344] LustreError: 206965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5661.241371] Lustre: server umount lustre-MDT0000 complete [ 5662.684684] LustreError: 203083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689955 with bad export cookie 3806374817876690735 [ 5662.688209] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.689779] LustreError: 203083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5662.826374] Lustre: server umount lustre-MDT0001 complete [ 5667.149626] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5668.346476] Key type lgssc unregistered [ 5668.485388] LNet: 207698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5668.490130] LNetError: 207698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5668.498364] LNet: Removed LNI 192.168.201.128@tcp [ 5668.856850] Key type .llcrypt unregistered [ 5668.858776] Key type ._llcrypt unregistered [ 5684.710953] Key type ._llcrypt registered [ 5684.712398] Key type .llcrypt registered [ 5684.753978] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5685.238480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5685.267262] alg: No test for adler32 (adler32-zlib) [ 5686.139376] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5686.238377] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5687.840139] Key type lgssc registered [ 5688.305469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5691.396160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5691.404800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.512455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5692.522403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5692.593686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.156967] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5697.463085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5697.506033] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5697.520247] Lustre: 209861:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5697.647727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5699.349377] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5701.564841] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.240267] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5705.905410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5705.994472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5706.082898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.208981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5708.237495] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5710.407974] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5712.086311] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5718.756520] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5728.424722] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5739.329273] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5752.412293] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5752.414977] Lustre: Skipped 1 previous similar message [ 5773.108446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5773.111640] Lustre: Skipped 1 previous similar message [ 5808.923433] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5808.927282] Lustre: Skipped 3 previous similar messages [ 5882.474873] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5882.478986] Lustre: Skipped 8 previous similar messages [ 5895.650335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.652550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.655885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.663730] Lustre: Skipped 1 previous similar message [ 5895.668863] Lustre: Skipped 1 previous similar message [ 5897.520081] LustreError: 213431:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5897.571747] Lustre: server umount lustre-OST0000 complete [ 5900.769758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.770615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.775621] Lustre: Skipped 1 previous similar message [ 5905.418417] LustreError: 213638:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5905.422927] LustreError: 213638:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5905.461419] Lustre: server umount lustre-MDT0000 complete [ 5907.089114] LustreError: 209853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690200 with bad export cookie 13307052384002094587 [ 5907.090071] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.094233] LustreError: 209853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5907.181604] LustreError: 213840:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5907.184870] LustreError: 213840:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5907.251321] Lustre: server umount lustre-MDT0001 complete [ 5911.663777] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5912.856376] Key type lgssc unregistered [ 5913.010462] LNet: 214370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5913.015929] LNetError: 214370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5913.024521] LNet: Removed LNI 192.168.201.128@tcp [ 5913.383350] Key type .llcrypt unregistered [ 5913.384533] Key type ._llcrypt unregistered [ 5923.662312] Key type ._llcrypt registered [ 5923.664127] Key type .llcrypt registered [ 5923.709364] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5924.062555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5924.079256] alg: No test for adler32 (adler32-zlib) [ 5924.950295] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5925.046366] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5926.640324] Key type lgssc registered [ 5927.086451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5930.069016] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5930.074465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.254944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.868939] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5936.258476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5936.435634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5938.104269] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5940.241376] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5941.732441] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5944.585485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5944.716660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.976289] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5949.254463] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5957.033812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5967.331479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5967.332072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.333591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5967.334727] LustreError: Skipped 1 previous similar message [ 5967.340460] Lustre: Skipped 1 previous similar message [ 5970.161127] LustreError: 218154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5970.204843] Lustre: server umount lustre-OST0000 complete [ 5972.450331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5972.450647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.451288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5972.451293] Lustre: Skipped 1 previous similar message [ 5972.462156] Lustre: Skipped 1 previous similar message [ 5977.570114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.573232] Lustre: Skipped 2 previous similar messages [ 5977.732386] LustreError: 218359:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5977.734903] LustreError: 218359:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5977.768553] Lustre: server umount lustre-MDT0000 complete [ 5979.190493] LustreError: 218067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690272 with bad export cookie 8133410384222290041 [ 5979.194300] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5979.195523] LustreError: 218067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5979.274090] LustreError: 218560:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5979.277215] LustreError: 218560:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5979.346655] Lustre: server umount lustre-MDT0001 complete [ 5983.553407] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5984.753471] Key type lgssc unregistered [ 5984.901435] LNet: 219093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5984.906285] LNetError: 219093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5984.914263] LNet: Removed LNI 192.168.201.128@tcp [ 5985.248201] Key type .llcrypt unregistered [ 5985.249534] Key type ._llcrypt unregistered [ 5994.778599] Key type ._llcrypt registered [ 5994.779991] Key type .llcrypt registered [ 5994.824164] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5995.378396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5995.389268] alg: No test for adler32 (adler32-zlib) [ 5996.266202] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5996.363714] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5997.944108] Key type lgssc registered [ 5998.343430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6266.177500] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 6268.266039] Key type lgssc unregistered [ 6268.479575] LNet: 221055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6268.485521] LNetError: 221055:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6268.497324] LNet: Removed LNI 192.168.201.128@tcp [ 6269.129308] Key type .llcrypt unregistered [ 6269.132537] Key type ._llcrypt unregistered [ 6284.409409] Key type ._llcrypt registered [ 6284.410880] Key type .llcrypt registered [ 6284.467898] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 6294.852248] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 6295.330191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6295.348445] alg: No test for adler32 (adler32-zlib) [ 6296.358819] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6296.519459] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 6298.128381] Key type lgssc registered [ 6298.873491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6316.220525] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5958 sec ========== 21:30:06 (1770690606) [ 6317.428675] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:30:08 (1770690608) === [ 6319.780099] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:30:10 (1770690610) === [ 6344.421244] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 6346.242973] Key type lgssc unregistered [ 6346.456074] LNet: 226209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6346.460252] LNetError: 226209:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6346.468583] LNet: Removed LNI 192.168.201.128@tcp [ 6346.950907] Key type .llcrypt unregistered [ 6346.953076] Key type ._llcrypt unregistered