[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1352171575 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.005000] x2apic enabled [ 0.005013] Switched APIC routing to physical x2apic. [ 0.006019] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010368] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012071] SELinux: Initializing. [ 0.013096] *** VALIDATE selinux *** [ 0.028043] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.036561] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.038070] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040121] *** VALIDATE tmpfs *** [ 0.041000] *** VALIDATE proc *** [ 0.043154] *** VALIDATE cgroup *** [ 0.044015] *** VALIDATE cgroup2 *** [ 0.045000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047077] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.049005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.050033] Spectre V2 : User space: Vulnerable [ 0.051012] Speculative Store Bypass: Vulnerable [ 0.057000] debug: unmapping init [mem 0xffffffff96459000-0xffffffff96460fff] [ 0.059413] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.061681] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.063028] ... version: 2 [ 0.064019] ... bit width: 48 [ 0.065020] ... generic registers: 4 [ 0.066019] ... value mask: 0000ffffffffffff [ 0.067019] ... max period: 00007fffffffffff [ 0.068018] ... fixed-purpose events: 3 [ 0.069019] ... event mask: 000000070000000f [ 0.070266] rcu: Hierarchical SRCU implementation. [ 0.072771] smp: Bringing up secondary CPUs ... [ 0.073641] x86: Booting SMP configuration: [ 0.074028] .... node #0, CPUs: #1 #2 #3 [ 0.083027] smp: Brought up 1 node, 4 CPUs [ 0.085019] smpboot: Max logical packages: 1 [ 0.086020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123178] node 0 deferred pages initialised in 33ms [ 0.128657] devtmpfs: initialized [ 0.129481] x86/mm: Memory block size: 128MB [ 0.132395] gcov: version magic: 0x41383552 [ 0.134508] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137490] pinctrl core: initialized pinctrl subsystem [ 0.140218] [ 0.140813] ************************************************************* [ 0.145014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148012] ** ** [ 0.151014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155015] ** ** [ 0.157102] ** This means that this kernel is built to expose internal ** [ 0.160014] ** IOMMU data structures, which may compromise security on ** [ 0.162019] ** your system. ** [ 0.164015] ** ** [ 0.167013] ** If you see this message and you are not debugging the ** [ 0.170014] ** kernel, report this immediately to your vendor! ** [ 0.173015] ** ** [ 0.176014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179012] ************************************************************* [ 0.185646] NET: Registered protocol family 16 [ 0.189711] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208149] cpuidle: using governor menu [ 0.209000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211459] PCI: Using configuration type 1 for base access [ 0.214147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.244215] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.245000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.254659] cryptd: max_cpu_qlen set to 1000 [ 0.259080] ACPI: Added _OSI(Module Device) [ 0.263016] ACPI: Added _OSI(Processor Device) [ 0.267016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.271072] ACPI: Added _OSI(Processor Aggregator Device) [ 0.279253] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.288475] ACPI: Interpreter enabled [ 0.289000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290011] ACPI: Using IOAPIC for interrupt routing [ 0.292628] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.297900] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.330000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.347023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.361076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.370000] acpiphp: Slot [2] registered [ 0.378125] acpiphp: Slot [5] registered [ 0.381166] acpiphp: Slot [6] registered [ 0.390427] acpiphp: Slot [7] registered [ 0.393158] acpiphp: Slot [8] registered [ 0.396174] acpiphp: Slot [9] registered [ 0.400149] acpiphp: Slot [10] registered [ 0.409059] acpiphp: Slot [3] registered [ 0.412131] acpiphp: Slot [4] registered [ 0.418106] acpiphp: Slot [11] registered [ 0.422141] acpiphp: Slot [12] registered [ 0.423115] acpiphp: Slot [13] registered [ 0.428132] acpiphp: Slot [14] registered [ 0.431126] acpiphp: Slot [15] registered [ 0.433115] acpiphp: Slot [16] registered [ 0.435123] acpiphp: Slot [17] registered [ 0.437128] acpiphp: Slot [18] registered [ 0.439134] acpiphp: Slot [19] registered [ 0.441139] acpiphp: Slot [20] registered [ 0.444165] acpiphp: Slot [21] registered [ 0.446130] acpiphp: Slot [22] registered [ 0.449156] acpiphp: Slot [23] registered [ 0.451243] acpiphp: Slot [24] registered [ 0.453122] acpiphp: Slot [25] registered [ 0.456175] acpiphp: Slot [26] registered [ 0.465149] acpiphp: Slot [27] registered [ 0.470032] acpiphp: Slot [28] registered [ 0.473129] acpiphp: Slot [29] registered [ 0.477142] acpiphp: Slot [30] registered [ 0.478143] acpiphp: Slot [31] registered [ 0.479000] PCI host bridge to bus 0000:00 [ 0.480023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.483031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.486027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.489132] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.492119] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.495023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.496000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.529009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.550236] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.568014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.573055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.581028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.583014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.588018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.589804] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.593000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.596042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.599959] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.603016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.623015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.633021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.645837] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.662017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.681019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.699017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.722000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.732015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.741015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.778020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.794076] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.801015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.809022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.830016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.846141] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.854016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.862015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.880017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.892015] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.898016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.906021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.920018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.940532] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.948015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.953015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.971015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.980000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.982617] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.985645] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.988615] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.991330] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.997278] iommu: Default domain type: Passthrough [ 0.999567] SCSI subsystem initialized [ 1.001190] ACPI: bus type USB registered [ 1.003141] usbcore: registered new interface driver usbfs [ 1.005104] usbcore: registered new interface driver hub [ 1.007096] usbcore: registered new device driver usb [ 1.009315] pps_core: LinuxPPS API ver. 1 registered [ 1.010011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.013060] PTP clock support registered [ 1.017170] EDAC MC: Ver: 3.0.0 [ 1.018259] PCI: Using ACPI for IRQ routing [ 1.021329] NetLabel: Initializing [ 1.024012] NetLabel: domain hash size = 128 [ 1.026020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.029083] NetLabel: unlabeled traffic allowed by default [ 1.035681] vgaarb: loaded [ 1.041262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.043010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.058244] clocksource: Switched to clocksource kvm-clock [ 1.281048] VFS: Disk quotas dquot_6.6.0 [ 1.282481] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.291395] *** VALIDATE ramfs *** [ 1.292644] *** VALIDATE hugetlbfs *** [ 1.294131] pnp: PnP ACPI init [ 1.299349] pnp: PnP ACPI: found 6 devices [ 1.331799] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.337637] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.349750] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.353909] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.359810] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.368374] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.377343] NET: Registered protocol family 2 [ 1.381176] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.389937] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.396227] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.405217] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.411801] TCP: Hash tables configured (established 65536 bind 65536) [ 1.420407] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.428128] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.433531] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.440772] NET: Registered protocol family 1 [ 1.447779] RPC: Registered named UNIX socket transport module. [ 1.449921] RPC: Registered udp transport module. [ 1.453284] RPC: Registered tcp transport module. [ 1.457601] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.464497] NET: Registered protocol family 44 [ 1.466164] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.470150] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.475162] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.481330] PCI: CLS 0 bytes, default 64 [ 1.484360] Unpacking initramfs... [ 6.489190] debug: unmapping init [mem 0xffff94ba3cc54000-0xffff94ba3ffbffff] [ 6.493847] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.496763] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.500526] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 10.766182] Initialise system trusted keyrings [ 10.787980] Key type blacklist registered [ 10.819236] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 10.893249] zbud: loaded [ 10.945204] *** VALIDATE nfs *** [ 10.946514] *** VALIDATE nfs4 *** [ 10.962583] pstore: using deflate compression [ 11.145108] Platform Keyring initialized [ 12.277369] NET: Registered protocol family 38 [ 12.279217] Key type asymmetric registered [ 12.289294] Asymmetric key parser 'x509' registered [ 12.294861] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 12.309165] io scheduler mq-deadline registered [ 12.310810] io scheduler kyber registered [ 12.313186] io scheduler bfq registered [ 12.323910] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 12.339216] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 12.353452] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 12.365473] ACPI: Power Button [PWRF] [ 12.389267] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 12.418477] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 12.470158] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 12.493641] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 12.555676] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 12.646038] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 12.734818] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 12.770252] Non-volatile memory driver v1.3 [ 12.785267] Linux agpgart interface v0.103 [ 13.040437] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 13.043313] vda: detected capacity change from 0 to 68624384 [ 13.285323] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 13.314652] vdb: detected capacity change from 0 to 1073741824 [ 13.460187] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 13.480941] vdc: detected capacity change from 0 to 2621440000 [ 13.554389] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 13.564463] vdd: detected capacity change from 0 to 2621440000 [ 13.633415] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 13.644618] vde: detected capacity change from 0 to 4294967296 [ 13.739535] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 13.775913] vdf: detected capacity change from 0 to 4294967296 [ 13.811425] libphy: Fixed MDIO Bus: probed [ 13.864245] usbcore: registered new interface driver usbserial_generic [ 13.882236] usbserial: USB Serial support registered for generic [ 13.891277] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 13.918073] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 13.924657] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 13.936764] mousedev: PS/2 mouse device common for all mice [ 13.953736] rtc_cmos 00:05: RTC can wake from S4 [ 13.963847] rtc_cmos 00:05: registered as rtc0 [ 13.979727] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 13.998569] intel_pstate: CPU model not supported [ 14.024600] hid: raw HID events driver (C) Jiri Kosina [ 14.025486] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 14.036100] usbcore: registered new interface driver usbhid [ 14.036108] usbhid: USB HID core driver [ 14.036255] drop_monitor: Initializing network drop monitor service [ 14.036385] Initializing XFRM netlink socket [ 14.036822] NET: Registered protocol family 10 [ 14.038950] Segment Routing with IPv6 [ 14.039035] NET: Registered protocol family 17 [ 14.043088] mpls_gso: MPLS GSO support [ 14.101409] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 14.179318] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 14.184448] RAS: Correctable Errors collector initialized. [ 14.187217] AVX version of gcm_enc/dec engaged. [ 14.189205] AES CTR mode by8 optimization enabled [ 14.537542] sched_clock: Marking stable (14537523335, 0)->(17718993139, -3181469804) [ 14.575134] registered taskstats version 1 [ 14.582055] Loading compiled-in X.509 certificates [ 14.586124] zswap: loaded using pool lzo/zbud [ 14.715667] Key type big_key registered [ 14.779532] Key type encrypted registered [ 14.781241] ima: No TPM chip found, activating TPM-bypass! [ 14.798056] ima: Allocated hash algorithm: sha1 [ 14.807491] ima: No architecture policies found [ 14.817094] evm: Initialising EVM extended attributes: [ 14.819093] evm: security.selinux [ 14.820203] evm: security.ima [ 14.830655] evm: security.capability [ 14.837319] evm: HMAC attrs: 0x1 [ 14.875771] rtc_cmos 00:05: setting system clock to 2026-01-22 09:10:29 UTC (1769073029) [ 14.895250] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 14.950416] debug: unmapping init [mem 0xffffffff96182000-0xffffffff96458fff] [ 14.979944] Write protecting the kernel read-only data: 28672k [ 15.005152] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 15.034944] debug: unmapping init [mem 0xffffffff95114000-0xffffffff951fffff] [ 15.275156] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.315552] systemd[1]: Detected virtualization kvm. [ 15.331983] systemd[1]: Detected architecture x86-64. [ 15.342575] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 15.573835] systemd[1]: No hostname configured. [ 15.579894] systemd[1]: Set hostname to . [ 15.589647] random: systemd: uninitialized urandom read (16 bytes read) [ 15.606190] systemd[1]: Initializing machine ID from random generator. [ 16.341387] random: systemd: uninitialized urandom read (16 bytes read) [ 16.347556] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 16.358928] random: systemd: uninitialized urandom read (16 bytes read) [ 16.368917] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 16.395182] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 20.724868] device-mapper: uevent: version 1.0.3 [ 20.727440] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] [ 22.088461] random: fast init done Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 22.259119] virtio_net virtio0 ens2: renamed from eth0 [ 22.753193] scsi host0: ata_piix [ 22.838627] scsi host1: ata_piix [ 22.841250] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 22.844736] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 28.404171] random: crng init done [ 28.409651] random: 7 urandom warning(s) missed due to ratelimiting [ 31.179371] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 33.486138] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 33.494467] hrtimer: interrupt took 12455371 ns [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 36.332147] printk: systemd: 26 output lines suppressed due to ratelimiting [ 37.555039] SELinux: Disabled at runtime. [ 37.710957] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 37.738071] systemd[1]: Detected virtualization kvm. [ 37.745553] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 39.740927] systemd[1]: initrd-switch-root.service: Succeeded. [ 39.745343] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 39.755492] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 39.761327] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 39.767777] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 39.778461] systemd[1]: Starting Journal Service... Starting Journal Service... [ 39.786745] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 40.274441] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 41.213581] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 42.797265] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.873710] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.166549] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 44.295776] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 50.097666] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 50.576695] NFS: Registering the id_resolver key type [ 50.580768] Key type id_resolver registered [ 50.582785] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg101-server login: [ 108.755059] libcfs: loading out-of-tree module taints kernel. [ 108.818684] Key type ._llcrypt registered [ 108.820406] Key type .llcrypt registered [ 108.899868] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 126.573716] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 127.696280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.712610] alg: No test for adler32 (adler32-zlib) [ 129.420803] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 130.366795] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 132.143231] Key type lgssc registered [ 134.111759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.777249] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.399840] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.334414] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.681580] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.196758] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.712945] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 204.603158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.668862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 204.684479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.965254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 205.996241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.100152] Lustre: lustre-MDT0000: new disk, initializing [ 206.175877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.200959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.441860] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.976159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.106213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 224.235028] Lustre: 6510:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.302235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 224.310267] Lustre: Skipped 1 previous similar message [ 224.431544] Lustre: lustre-MDT0001: new disk, initializing [ 224.545370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.579588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.584354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 229.250533] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.011496] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 243.638234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 243.722544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.919479] Lustre: lustre-OST0000: new disk, initializing [ 243.928071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.987284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.771434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.777927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.851841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.231650] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.448067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 264.532091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 264.670354] Lustre: lustre-OST0001: new disk, initializing [ 264.677313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.755809] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.540929] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.944866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.961225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.077834] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.294881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 294.748957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.555474] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.845420] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing yml_node [ 321.988476] Lustre: DEBUG MARKER: Client: 2.17.50.4 [ 326.428529] Lustre: DEBUG MARKER: MDS: 2.17.50.4 [ 330.397565] Lustre: DEBUG MARKER: OSS: 2.17.50.4 [ 332.819992] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 22 04:15:45 EST 2026 [ 358.477599] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 360.592098] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 378.357142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.365812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.393161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.895955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.900903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.713488] LustreError: 12217:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 384.787526] Lustre: server umount lustre-MDT0000 complete [ 390.138768] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.170436] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 390.239459] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073404 with bad export cookie 11689920251474487740 [ 390.249487] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.261479] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 395.233958] LustreError: 12110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.237703] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.240833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.240843] Lustre: Skipped 3 previous similar messages [ 395.272574] LustreError: 12110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 395.350846] Lustre: Skipped 3 previous similar messages [ 396.626599] LustreError: 12421:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 396.642111] LustreError: 12421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 397.424997] Lustre: server umount lustre-MDT0001 complete [ 402.656597] LustreError: 12624:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 402.667378] LustreError: 12624:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 403.047295] Lustre: server umount lustre-OST0000 complete [ 409.405416] LustreError: 12826:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 409.421763] LustreError: 12826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 409.635204] Lustre: server umount lustre-OST0001 complete [ 420.668585] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 434.692214] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 447.623904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 457.220624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 464.831587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 473.837468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 483.992533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 484.087912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.533976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 484.624563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 484.817617] Lustre: lustre-MDT0000: new disk, initializing [ 485.096469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.149698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 491.118631] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 504.932443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 505.038703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 505.244833] Lustre: 15015:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 505.391815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 505.395164] Lustre: Skipped 1 previous similar message [ 505.464840] Lustre: lustre-MDT0001: new disk, initializing [ 505.587748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.671520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 505.692383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 511.501266] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 521.972112] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 526.542626] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 537.353795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 537.518231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 538.136181] Lustre: lustre-OST0000: new disk, initializing [ 538.144774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 538.278098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.174986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 540.181949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 540.413993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 547.310890] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 557.810454] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 563.585321] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 563.845456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 570.032475] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 570.369885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 576.492548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 576.495180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.505873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.505886] Lustre: Skipped 1 previous similar message [ 576.512191] LustreError: Skipped 1 previous similar message [ 576.553151] Lustre: Skipped 1 previous similar message [ 578.766815] LustreError: 17954:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 578.777948] LustreError: 17954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 579.225463] Lustre: server umount lustre-OST0000 complete [ 586.722259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.775821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 586.791000] Lustre: Skipped 2 previous similar messages [ 590.160053] LustreError: 18155:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 590.165414] LustreError: 18155:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 590.362731] Lustre: server umount lustre-MDT0000 complete [ 591.840878] LustreError: 15022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.856766] LustreError: 15022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 596.562810] LustreError: 15007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073611 with bad export cookie 11689920251474489266 [ 596.575126] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.049346] Lustre: server umount lustre-MDT0001 complete [ 603.090905] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:20:15 (1769073615) [ 612.723073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.118660] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.198775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.467744] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.504025] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 627.681833] LustreError: 18884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.701878] LustreError: 18884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 631.343803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 637.952391] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 645.396120] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 651.003345] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 660.319830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 669.365743] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 678.130829] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 693.728329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 693.740126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.754676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 693.758140] Lustre: Skipped 2 previous similar messages [ 693.774807] Lustre: Skipped 1 previous similar message [ 699.139806] LustreError: 21158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 699.149517] LustreError: 21158:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 699.312608] Lustre: server umount lustre-OST0000 complete [ 708.577026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 710.830338] Lustre: server umount lustre-MDT0000 complete [ 714.210196] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.228236] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 715.993775] LustreError: 18863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073730 with bad export cookie 11689920251474490155 [ 716.003387] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 716.015963] LustreError: 18863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 716.654583] Lustre: server umount lustre-MDT0001 complete [ 730.424650] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 734.755526] Key type lgssc unregistered [ 735.261971] LNet: 22097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.288074] LNetError: 22097:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.313783] LNet: Removed LNI 192.168.201.101@tcp [ 736.568187] Key type .llcrypt unregistered [ 736.570485] Key type ._llcrypt unregistered [ 769.156756] Key type ._llcrypt registered [ 769.159251] Key type .llcrypt registered [ 769.374243] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 771.282820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 771.410638] alg: No test for adler32 (adler32-zlib) [ 772.796378] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 773.237123] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 775.063256] Key type lgssc registered [ 777.297207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.768239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.799678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.603390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.869929] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 800.242288] LustreError: 23494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.272402] LustreError: 23494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 805.347392] LustreError: 23494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.269970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 806.900448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 812.995749] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 820.604609] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 824.922495] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 834.201370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 834.541353] LustreError: 25096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.773110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.200342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 843.654289] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 852.594942] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 876.008922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 876.026609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.060735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.041925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.045341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.053602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.539634] LustreError: 26186:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 878.792640] Lustre: server umount lustre-OST0000 complete [ 887.272234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.295721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.296810] Lustre: Skipped 1 previous similar message [ 887.334476] Lustre: Skipped 2 previous similar messages [ 890.106289] LustreError: 26387:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 890.117716] LustreError: 26387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 890.663953] Lustre: server umount lustre-MDT0000 complete [ 892.392197] LustreError: 25104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.433485] LustreError: 25104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 895.966434] LustreError: 24215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073910 with bad export cookie 10420246708027754872 [ 895.968864] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.987125] LustreError: 24215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 896.166102] LustreError: 26589:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 896.171229] LustreError: 26589:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 896.358882] Lustre: server umount lustre-MDT0001 complete [ 911.759216] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 915.255776] Key type lgssc unregistered [ 915.689615] LNet: 27124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.700180] LNetError: 27124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.726714] LNet: Removed LNI 192.168.201.101@tcp [ 916.644170] Key type .llcrypt unregistered [ 916.646297] Key type ._llcrypt unregistered [ 943.488651] Key type ._llcrypt registered [ 943.502534] Key type .llcrypt registered [ 943.672353] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 945.199479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.237336] alg: No test for adler32 (adler32-zlib) [ 946.478465] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 946.836785] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 948.632183] Key type lgssc registered [ 950.298663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.043241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.075629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.876322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.540461] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 972.258406] LustreError: 28522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.287841] LustreError: 28522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 977.378817] LustreError: 28522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.132344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 978.659534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.514335] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 992.025373] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 996.790793] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1012.805075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1013.274552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.435057] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1027.507100] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1031.764530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1038.817506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1038.827807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.864550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.889170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.900223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.911427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.963464] Lustre: Skipped 1 previous similar message [ 1044.820951] LustreError: 31117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1045.029127] Lustre: server umount lustre-OST0000 complete [ 1052.132073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.141115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.153317] Lustre: Skipped 1 previous similar message [ 1056.525949] LustreError: 31320:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1056.539982] LustreError: 31320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1056.862864] Lustre: server umount lustre-MDT0000 complete [ 1057.258833] LustreError: 28523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.293637] LustreError: 28523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1062.369193] LustreError: 28517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.409786] LustreError: 28517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1062.903106] LustreError: 28503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074077 with bad export cookie 10518483924956570816 [ 1062.906703] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.909810] LustreError: 28503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1063.077235] LustreError: 31524:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1063.088282] LustreError: 31524:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1063.450870] Lustre: server umount lustre-MDT0001 complete [ 1077.781597] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1081.372269] Key type lgssc unregistered [ 1081.656850] LNet: 32057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.665586] LNetError: 32057:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.687401] LNet: Removed LNI 192.168.201.101@tcp [ 1082.815451] Key type .llcrypt unregistered [ 1082.824065] Key type ._llcrypt unregistered [ 1112.501070] Key type ._llcrypt registered [ 1112.504484] Key type .llcrypt registered [ 1112.624566] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1114.715489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1114.803136] alg: No test for adler32 (adler32-zlib) [ 1116.277902] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1116.635417] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1118.415610] Key type lgssc registered [ 1120.416358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.763441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.787422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.169151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.282049] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1142.753905] LustreError: 33459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.778457] LustreError: 33459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1147.872648] LustreError: 33459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.914985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1150.490864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.503336] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1163.091229] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1167.772903] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1175.652261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.045863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.099092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1183.342733] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1190.861694] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1208.808196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.816284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.826531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1209.186663] LustreError: 35729:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1209.373081] Lustre: server umount lustre-OST0000 complete [ 1214.944718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.946283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.970619] Lustre: Skipped 1 previous similar message [ 1214.983522] Lustre: Skipped 1 previous similar message [ 1220.065725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.072931] Lustre: Skipped 1 previous similar message [ 1220.271900] LustreError: 35930:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1220.301974] LustreError: 35930:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1220.436546] Lustre: server umount lustre-MDT0000 complete [ 1225.198973] LustreError: 33454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.225595] LustreError: 33454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1225.260219] LustreError: 33440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074239 with bad export cookie 5396841367899486597 [ 1225.271635] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.465985] LustreError: 36132:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1234.475499] LustreError: 36132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1234.739973] Lustre: server umount lustre-MDT0001 complete [ 1249.795495] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1253.166345] Key type lgssc unregistered [ 1253.610744] LNet: 36668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.623967] LNetError: 36668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.641870] LNet: Removed LNI 192.168.201.101@tcp [ 1254.571609] Key type .llcrypt unregistered [ 1254.574830] Key type ._llcrypt unregistered [ 1281.037444] Key type ._llcrypt registered [ 1281.040902] Key type .llcrypt registered [ 1281.168992] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1282.832672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.906858] alg: No test for adler32 (adler32-zlib) [ 1284.119963] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1284.424199] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1286.185125] Key type lgssc registered [ 1287.321664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1296.335405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1296.366396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.127735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.599410] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1307.626060] LustreError: 38063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.668964] LustreError: 38063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1312.738872] LustreError: 38063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.388555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1313.891619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1318.839600] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1325.603650] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.980543] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1337.566543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1338.046527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.165359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1344.776972] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1350.616591] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1358.422527] Lustre: lustre-OST0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 1358.816427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1358.831280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.849152] Lustre: Skipped 1 previous similar message [ 1363.459250] LustreError: 40311:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1363.519128] LustreError: 39941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.832045] Lustre: server umount lustre-OST0000 complete [ 1384.417386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.452714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.464668] Lustre: Skipped 2 previous similar messages [ 1389.536273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.545232] Lustre: Skipped 1 previous similar message [ 1390.694282] LustreError: 40690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1390.700087] LustreError: 40690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1390.895831] Lustre: server umount lustre-MDT0000 complete [ 1394.664136] LustreError: 38064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.694117] LustreError: 38064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1395.276119] LustreError: 38043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074409 with bad export cookie 17570835656899335357 [ 1395.284914] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.293587] LustreError: 38043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1401.599983] LustreError: 40892:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1401.605934] LustreError: 40892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1401.828350] Lustre: server umount lustre-MDT0001 complete [ 1413.528434] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1416.654866] Key type lgssc unregistered [ 1417.021859] LNet: 41425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.028669] LNetError: 41425:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.043241] LNet: Removed LNI 192.168.201.101@tcp [ 1418.076483] Key type .llcrypt unregistered [ 1418.078733] Key type ._llcrypt unregistered [ 1442.575289] Key type ._llcrypt registered [ 1442.577602] Key type .llcrypt registered [ 1442.667231] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1444.009981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.038227] alg: No test for adler32 (adler32-zlib) [ 1445.242730] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1445.543626] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1447.295231] Key type lgssc registered [ 1448.621576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.332669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.346573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.031450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.986476] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1468.388650] LustreError: 42823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.420492] LustreError: 42823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1473.519082] LustreError: 42823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.879193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1475.671874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.985798] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1488.448871] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.531837] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1501.983748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.453869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.517770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1509.243590] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1515.979739] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1529.460416] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 1531.367631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.371351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.385444] Lustre: Skipped 2 previous similar messages [ 1531.397466] Lustre: Skipped 2 previous similar messages [ 1533.729804] LustreError: 45071:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1534.012265] Lustre: server umount lustre-MDT0000 complete [ 1534.553943] LustreError: 42818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.093166] LustreError: 42803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074553 with bad export cookie 2376616809639290838 [ 1539.109279] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.115750] LustreError: 42803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1539.674137] LustreError: 42819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.712286] LustreError: 42819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1539.714908] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.1@tcp (stopping) [ 1541.601055] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.613692] LustreError: 42819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.614647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1544.635051] LustreError: 42819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1544.640077] Lustre: Skipped 1 previous similar message [ 1545.364927] LustreError: 45274:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1545.373187] LustreError: 45274:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1545.707558] Lustre: server umount lustre-MDT0001 complete [ 1561.424956] LustreError: 45499:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1561.445132] LustreError: 45499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1561.694434] Lustre: server umount lustre-OST0000 complete [ 1580.276116] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1584.423680] Key type lgssc unregistered [ 1585.104638] LNet: 46336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.123194] LNetError: 46336:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.152047] LNet: Removed LNI 192.168.201.101@tcp [ 1587.411526] Key type .llcrypt unregistered [ 1587.416310] Key type ._llcrypt unregistered [ 1618.348848] Key type ._llcrypt registered [ 1618.352330] Key type .llcrypt registered [ 1618.548513] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1619.603378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.755117] alg: No test for adler32 (adler32-zlib) [ 1620.924645] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1621.368019] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1623.079302] Key type lgssc registered [ 1625.058367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1635.382711] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1635.412744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1651.104898] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1656.223227] LustreError: 47705:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 1682.975360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.657507] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1696.852709] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1862.909423] LustreError: 48413:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1862.991212] Lustre: server umount lustre-OST0000 complete [ 1880.028819] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1882.341644] Key type lgssc unregistered [ 1882.593699] LNet: 49252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.603160] LNetError: 49252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.623346] LNet: Removed LNI 192.168.201.101@tcp [ 1883.408577] Key type .llcrypt unregistered [ 1883.414802] Key type ._llcrypt unregistered [ 1907.386546] Key type ._llcrypt registered [ 1907.389163] Key type .llcrypt registered [ 1907.552710] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1908.762627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.932212] alg: No test for adler32 (adler32-zlib) [ 1910.396495] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1910.803387] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1912.679955] Key type lgssc registered [ 1914.115945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1922.000523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1922.022310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.623898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.452802] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1933.794923] LustreError: 50651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.834724] LustreError: 50651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1937.868355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1938.295514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.346347] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1947.697251] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1950.790182] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1958.112799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1958.491524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.663286] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1968.786866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1969.436547] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1973.216082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.219044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.228721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.242372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.242503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.265837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.437112] LustreError: 52895:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1977.646850] Lustre: server umount lustre-OST0000 complete [ 1984.485985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.487680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.502478] Lustre: Skipped 1 previous similar message [ 1984.520325] Lustre: Skipped 2 previous similar messages [ 1987.831818] LustreError: 53096:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1987.838451] LustreError: 53096:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1987.997467] Lustre: server umount lustre-MDT0000 complete [ 1989.601747] LustreError: 50651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.623587] LustreError: 50651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1991.428548] LustreError: 50632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075006 with bad export cookie 3842992601941307817 [ 1991.437588] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.581862] LustreError: 53300:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1991.589019] LustreError: 53300:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1991.943992] Lustre: server umount lustre-MDT0001 complete [ 2002.901341] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2005.690322] Key type lgssc unregistered [ 2006.002511] LNet: 53832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.014240] LNetError: 53832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.035368] LNet: Removed LNI 192.168.201.101@tcp [ 2006.896237] Key type .llcrypt unregistered [ 2006.898115] Key type ._llcrypt unregistered [ 2028.000846] Key type ._llcrypt registered [ 2028.002636] Key type .llcrypt registered [ 2028.092084] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2029.249097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.374439] alg: No test for adler32 (adler32-zlib) [ 2030.568623] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2030.828196] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2032.487169] Key type lgssc registered [ 2033.688984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2041.924381] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2041.940196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2057.439841] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2062.559308] LustreError: 55198:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 2089.247244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.623993] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2101.941774] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2110.090758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.219618] LustreError: 54533:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003684ce99 x1855009634388224/t0(0) o250->MGC192.168.201.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.582923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.637274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2115.425969] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2116.072556] LustreError: 55224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.088263] LustreError: 55224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2121.195276] LustreError: 55224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.264478] LustreError: 55224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.285573] LustreError: 55224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2124.721278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2125.056336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.746651] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2135.741360] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2140.414367] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2145.783109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.786395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2145.801619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2145.827921] Lustre: Skipped 1 previous similar message [ 2148.722292] LustreError: 57476:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2149.058332] Lustre: server umount lustre-OST0000 complete [ 2176.480762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.483201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.495266] Lustre: Skipped 1 previous similar message [ 2176.497772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.531879] Lustre: Skipped 1 previous similar message [ 2180.525167] LustreError: 57855:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2180.535786] LustreError: 57855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2180.666343] Lustre: server umount lustre-MDT0000 complete [ 2181.606956] LustreError: 57594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.619781] LustreError: 57594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2184.952639] LustreError: 55205:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075199 with bad export cookie 3235821642682752698 [ 2184.963852] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.969644] LustreError: 55205:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2191.230557] LustreError: 58057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2191.233932] LustreError: 58057:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2191.375321] Lustre: server umount lustre-MDT0001 complete [ 2207.360857] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2211.444695] Key type lgssc unregistered [ 2211.756966] LNet: 58592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.768753] LNetError: 58592:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.784165] LNet: Removed LNI 192.168.201.101@tcp [ 2212.626739] Key type .llcrypt unregistered [ 2212.628550] Key type ._llcrypt unregistered [ 2237.030196] Key type ._llcrypt registered [ 2237.032134] Key type .llcrypt registered [ 2237.141577] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2238.268641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.368898] alg: No test for adler32 (adler32-zlib) [ 2239.552729] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2239.884427] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2241.615428] Key type lgssc registered [ 2242.833707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2250.951599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2250.969862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.479814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.285519] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2263.031263] LustreError: 59987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.050030] LustreError: 59987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2265.680639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2266.128818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2271.022826] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2276.878845] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2280.673616] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2289.039838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2289.480850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.150490] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2297.735866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2303.389814] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2323.424916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.447145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.461424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.478543] Lustre: Skipped 1 previous similar message [ 2328.180089] LustreError: 62255:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2328.493531] Lustre: server umount lustre-OST0000 complete [ 2332.644738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2332.655430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.667953] Lustre: Skipped 1 previous similar message [ 2332.672391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.665580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.704425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.721097] Lustre: Skipped 1 previous similar message [ 2338.752947] LustreError: 62458:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2338.763376] LustreError: 62458:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2338.790281] LustreError: 59988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.822148] LustreError: 59988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2339.179919] Lustre: server umount lustre-MDT0000 complete [ 2343.374980] LustreError: 59969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075357 with bad export cookie 18191463063696585653 [ 2343.383338] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.391446] LustreError: 59969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2343.747482] LustreError: 62660:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2343.767851] LustreError: 62660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2344.131548] Lustre: server umount lustre-MDT0001 complete [ 2359.635797] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2362.978546] Key type lgssc unregistered [ 2363.345869] LNet: 63194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.357679] LNetError: 63194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.383871] LNet: Removed LNI 192.168.201.101@tcp [ 2364.472200] Key type .llcrypt unregistered [ 2364.475875] Key type ._llcrypt unregistered [ 2392.812965] Key type ._llcrypt registered [ 2392.815428] Key type .llcrypt registered [ 2392.899463] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2394.155734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.184074] alg: No test for adler32 (adler32-zlib) [ 2395.204580] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2395.399411] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2397.111189] Key type lgssc registered [ 2398.263517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2408.384577] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:50:20 (1769075420) [ 2416.931894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2416.956607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.512484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.815731] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2428.900537] LustreError: 64882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.927742] LustreError: 64882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2433.367826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2433.761661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.467045] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2446.025129] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2450.965122] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2459.063902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2459.688620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.809908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2467.051582] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2473.021298] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2482.399758] Lustre: Failing over lustre-MDT0000 [ 2482.584884] LustreError: 67131:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2484.203530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.221023] Lustre: Skipped 1 previous similar message [ 2484.221343] LustreError: 64878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.252140] LustreError: 64878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2484.642634] LustreError: 67131:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2484.647920] LustreError: 67131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2484.777869] Lustre: server umount lustre-MDT0000 complete [ 2485.343327] LustreError: 64879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.328584] LustreError: 64879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.368987] LustreError: 64879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2494.439543] LustreError: 65996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.451568] LustreError: 65996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2495.159720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.300901] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.465432] Lustre: *** cfs_fail_loc=135, val=0*** [ 2495.556460] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (not set up) [ 2495.591517] LustreError: 67641:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2495.600251] LustreError: 67641:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -2 [ 2495.616531] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2495.616531] [ 2495.629591] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2495.642145] LustreError: 67629:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2495.652038] LustreError: 67629:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2495.658931] LustreError: 67629:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2495.664549] LustreError: 67629:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2495.685761] Lustre: server umount lustre-MDT0000 complete [ 2495.689956] LustreError: 67629:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2503.513239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.698781] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.733489] LustreError: 64875:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2504.180942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.237099] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2505.782712] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2508.593228] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2509.300679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2509.326934] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2509.376840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2519.530747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.560125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.569709] Lustre: Skipped 1 previous similar message [ 2524.650169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.665290] Lustre: Skipped 1 previous similar message [ 2524.899187] LustreError: 68450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2524.901932] LustreError: 68450:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2525.204862] Lustre: server umount lustre-OST0000 complete [ 2529.761124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.784249] Lustre: Skipped 1 previous similar message [ 2529.792573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.883273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.903033] Lustre: Skipped 2 previous similar messages [ 2535.696804] LustreError: 68652:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2535.713490] LustreError: 68652:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2536.196423] Lustre: server umount lustre-MDT0000 complete [ 2540.006693] LustreError: 65596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.029682] LustreError: 65596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2541.582279] LustreError: 64865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075556 with bad export cookie 135458449979491512 [ 2541.606111] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2541.609876] LustreError: 64865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2541.823206] LustreError: 68856:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2541.831702] LustreError: 68856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2542.094934] Lustre: server umount lustre-MDT0001 complete [ 2553.729295] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2557.218283] Key type lgssc unregistered [ 2557.542982] LNet: 69388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.548439] LNetError: 69388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.560550] LNet: Removed LNI 192.168.201.101@tcp [ 2558.483846] Key type .llcrypt unregistered [ 2558.485657] Key type ._llcrypt unregistered [ 2584.593464] Key type ._llcrypt registered [ 2584.595415] Key type .llcrypt registered [ 2584.729177] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2586.327929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2586.399829] alg: No test for adler32 (adler32-zlib) [ 2587.593528] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2588.011605] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2589.712758] Key type lgssc registered [ 2590.916869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2601.156587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2601.187261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.671967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.895764] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2613.218447] LustreError: 70800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.252502] LustreError: 70800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2616.839438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2617.247750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2622.537933] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2628.627709] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2632.787265] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2639.545165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.025436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.069564] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2648.260685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2653.095394] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2660.907108] Lustre: Failing over lustre-MDT0000 [ 2660.987112] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 2661.224733] LustreError: 73049:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2661.591718] Lustre: server umount lustre-MDT0000 complete [ 2663.393467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2663.400161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.420895] LustreError: 71512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.916093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.071743] LustreError: 70794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.104732] LustreError: 70794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2669.039091] LustreError: 70794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.068299] LustreError: 70794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2673.040973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.249885] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2673.489329] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2673.630258] LustreError: 73559:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2673.635493] LustreError: 73559:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -9 [ 2673.649147] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2673.649147] [ 2673.655170] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2673.679702] LustreError: 73547:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2673.694732] LustreError: 73547:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2673.712040] LustreError: 73547:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2673.716445] LustreError: 73547:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2673.739898] Lustre: server umount lustre-MDT0000 complete [ 2673.743959] LustreError: 73547:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2674.146864] LustreError: 70800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.166144] LustreError: 70800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2682.051956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.289466] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.305312] LustreError: 73906:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.201.101@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000447ed41f) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2682.333560] LustreError: 70792:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2682.481308] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (not set up) [ 2682.697872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.806731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2686.528470] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2687.827540] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2687.989106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2688.009552] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2688.053319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2698.218757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.221766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2698.244997] Lustre: Skipped 2 previous similar messages [ 2698.276762] Lustre: Skipped 1 previous similar message [ 2701.260091] LustreError: 74371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2701.277216] LustreError: 74371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2701.455282] Lustre: server umount lustre-OST0000 complete [ 2708.454714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2708.458232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.462778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.481043] Lustre: Skipped 1 previous similar message [ 2708.514752] Lustre: Skipped 1 previous similar message [ 2713.572071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.227497] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2720.311248] LustreError: 74572:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2720.314694] LustreError: 74572:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2720.412833] Lustre: server umount lustre-MDT0000 complete [ 2722.789258] LustreError: 70799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.812230] LustreError: 70799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2724.975441] LustreError: 71520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075739 with bad export cookie 3381213367249602306 [ 2724.978397] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.992594] LustreError: 71520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2725.156861] LustreError: 74775:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2725.164839] LustreError: 74775:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2725.531474] Lustre: server umount lustre-MDT0001 complete [ 2737.484481] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2741.081060] Key type lgssc unregistered [ 2741.559261] LNet: 75307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2741.573123] LNetError: 75307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2741.595078] LNet: Removed LNI 192.168.201.101@tcp [ 2742.553231] Key type .llcrypt unregistered [ 2742.555148] Key type ._llcrypt unregistered [ 2756.349442] Key type ._llcrypt registered [ 2756.354564] Key type .llcrypt registered [ 2756.510453] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2756.881502] Key type .llcrypt unregistered [ 2756.888128] Key type ._llcrypt unregistered [ 2787.586207] Key type ._llcrypt registered [ 2787.589324] Key type .llcrypt registered [ 2787.780621] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2789.865118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2789.905693] alg: No test for adler32 (adler32-zlib) [ 2791.052467] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2791.372788] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2793.095226] Key type lgssc registered [ 2794.190309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2802.257042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2802.277241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.961045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.166360] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2813.418930] LustreError: 77174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.439797] LustreError: 77174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2817.956249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2818.230647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2822.911255] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2829.420639] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2833.950984] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2843.346765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2843.868815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.026126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2850.721407] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2857.430707] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2873.831218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.835191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2873.856638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.868689] LustreError: 79442:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2878.983255] LustreError: 78772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.141357] Lustre: server umount lustre-OST0000 complete [ 2885.095152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2885.104863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.120198] Lustre: Skipped 1 previous similar message [ 2885.141431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.152039] Lustre: Skipped 1 previous similar message [ 2889.185515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.187621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.216440] Lustre: Skipped 1 previous similar message [ 2890.501068] LustreError: 79644:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2890.508281] LustreError: 79644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2890.662632] Lustre: server umount lustre-MDT0000 complete [ 2894.311691] LustreError: 77174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.341993] LustreError: 77174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2895.484347] LustreError: 77153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075910 with bad export cookie 11456280097133704870 [ 2895.496456] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2895.505548] LustreError: 77153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2895.625951] LustreError: 79847:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2895.632310] LustreError: 79847:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2895.808290] Lustre: server umount lustre-MDT0001 complete [ 2908.154621] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2911.201970] Key type lgssc unregistered [ 2911.506571] LNet: 80379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2911.515576] LNetError: 80379:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2911.535696] LNet: Removed LNI 192.168.201.101@tcp [ 2912.620516] Key type .llcrypt unregistered [ 2912.633203] Key type ._llcrypt unregistered [ 2936.327481] Key type ._llcrypt registered [ 2936.332184] Key type .llcrypt registered [ 2936.483893] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2937.900352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2937.917429] alg: No test for adler32 (adler32-zlib) [ 2939.107381] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2939.558255] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2941.375205] Key type lgssc registered [ 2942.884552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2951.505246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2951.528227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2953.276280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.520784] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2962.406138] LustreError: 81772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.426608] LustreError: 81772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2967.521205] LustreError: 81772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.897713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2969.247786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2973.736679] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2980.442364] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2985.359741] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2993.781622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2994.144031] LustreError: 83375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.572254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.739102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3001.839591] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3008.261133] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3018.210871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.235759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.246107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3018.253628] Lustre: Skipped 1 previous similar message [ 3020.933186] LustreError: 84022:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3021.033576] Lustre: server umount lustre-OST0000 complete [ 3025.380609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.399135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.411052] Lustre: Skipped 1 previous similar message [ 3025.422778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.452319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.453915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.568154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.577380] Lustre: Skipped 2 previous similar messages [ 3037.665417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.671531] Lustre: Skipped 1 previous similar message [ 3039.199185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3039.278586] LustreError: 84223:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3039.292300] LustreError: 84223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3039.442737] Lustre: server umount lustre-MDT0000 complete [ 3042.784880] LustreError: 82485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.802153] LustreError: 82485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3043.276500] LustreError: 81752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076057 with bad export cookie 9696340421415222884 [ 3043.292179] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.294810] LustreError: 81752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3043.467292] LustreError: 84426:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3043.471194] LustreError: 84426:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3043.694775] Lustre: server umount lustre-MDT0001 complete [ 3056.208346] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3058.680217] Key type lgssc unregistered [ 3059.029549] LNet: 84971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3059.038895] LNetError: 84971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3059.064995] LNet: Removed LNI 192.168.201.101@tcp [ 3059.956228] Key type .llcrypt unregistered [ 3059.963601] Key type ._llcrypt unregistered [ 3083.422875] Key type ._llcrypt registered [ 3083.425333] Key type .llcrypt registered [ 3083.510327] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3084.556642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3084.724032] alg: No test for adler32 (adler32-zlib) [ 3085.884886] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3086.074500] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3087.755358] Key type lgssc registered [ 3089.179637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3098.123206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3098.145288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.634718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.141703] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3109.861646] LustreError: 86364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.885474] LustreError: 86364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3112.631867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3112.980326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3116.369105] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3121.423873] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3124.757775] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3132.510452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.877927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.008716] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3145.168135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3147.146650] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3158.499024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3158.507960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3158.521137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.036338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3160.040116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.068328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.240156] LustreError: 88612:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3160.359486] Lustre: server umount lustre-OST0000 complete [ 3165.671646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.685724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.698206] Lustre: Skipped 1 previous similar message [ 3170.474356] LustreError: 88812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3170.488173] LustreError: 88812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3170.616830] Lustre: server umount lustre-MDT0000 complete [ 3170.790430] LustreError: 86363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.808076] LustreError: 86363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3175.279451] LustreError: 86344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076189 with bad export cookie 18088470278947202553 [ 3175.286488] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.298568] LustreError: 86344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3175.509106] LustreError: 89014:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3175.523810] LustreError: 89014:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3175.733944] Lustre: server umount lustre-MDT0001 complete [ 3188.419528] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3191.266565] Key type lgssc unregistered [ 3191.655904] LNet: 89547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3191.662975] LNetError: 89547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3192.680691] LNet: Removed LNI 192.168.201.101@tcp [ 3194.015328] Key type .llcrypt unregistered [ 3194.020240] Key type ._llcrypt unregistered [ 3220.992641] Key type ._llcrypt registered [ 3220.994065] Key type .llcrypt registered [ 3221.112935] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3222.680803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3222.697204] alg: No test for adler32 (adler32-zlib) [ 3223.928625] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3224.245650] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3225.977084] Key type lgssc registered [ 3228.101555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3241.419756] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3241.428529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3256.994215] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3262.112331] LustreError: 90917:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 3288.799916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.061105] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3302.039286] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3309.500165] LustreError: 91776:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3309.588106] Lustre: server umount lustre-OST0000 complete [ 3319.429491] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:05:32 (1769076332) [ 3328.413397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.882408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.812708] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3334.117693] LustreError: 92446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.144504] LustreError: 92446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3339.239334] LustreError: 92446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.362115] LustreError: 92446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.236296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3345.747238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3350.700174] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3357.583179] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.602074] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3372.686717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3372.943597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.014340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3379.691894] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3386.435513] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3404.775433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3404.792357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.805315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.818731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.823796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.824808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.229671] LustreError: 94856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3408.243047] LustreError: 94856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3408.626226] Lustre: server umount lustre-OST0000 complete [ 3417.057995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.083863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3417.086535] Lustre: Skipped 1 previous similar message [ 3419.710668] LustreError: 95057:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3419.715843] LustreError: 95057:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3419.950138] Lustre: server umount lustre-MDT0000 complete [ 3422.178332] LustreError: 92447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.196115] LustreError: 92447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3424.536992] LustreError: 92426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076439 with bad export cookie 30329128867432441 [ 3424.548690] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.551435] LustreError: 92426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3424.819952] LustreError: 95259:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3424.835518] LustreError: 95259:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3425.294981] Lustre: server umount lustre-MDT0001 complete [ 3440.567404] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3444.181415] Key type lgssc unregistered [ 3444.568641] LNet: 95794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3444.578363] LNetError: 95794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3444.600430] LNet: Removed LNI 192.168.201.101@tcp [ 3445.615250] Key type .llcrypt unregistered [ 3445.625222] Key type ._llcrypt unregistered [ 3474.048965] Key type ._llcrypt registered [ 3474.054583] Key type .llcrypt registered [ 3474.186841] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3475.620682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3475.689120] alg: No test for adler32 (adler32-zlib) [ 3476.769985] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3476.986088] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3478.640648] Key type lgssc registered [ 3480.049096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3490.418412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3490.456923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.894470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.080494] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3502.069690] LustreError: 97188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.113621] LustreError: 97188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3507.189476] LustreError: 97188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.775725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3510.240525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.353119] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3522.934963] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3527.398856] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3535.466776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3536.075663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.144758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3543.778626] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3550.986679] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3566.079780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3566.092422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.121616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.157475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.169369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.171828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.197570] Lustre: Skipped 1 previous similar message [ 3571.724342] LustreError: 99459:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3572.148983] Lustre: server umount lustre-OST0000 complete [ 3576.800301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3576.812622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.838048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.387558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.388241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.428197] Lustre: Skipped 1 previous similar message [ 3582.913764] LustreError: 99662:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3582.923470] LustreError: 99662:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3583.123566] Lustre: server umount lustre-MDT0000 complete [ 3585.514332] LustreError: 97187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.534653] LustreError: 97187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3587.383587] LustreError: 97169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076602 with bad export cookie 15179756098138674385 [ 3587.384602] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.406665] LustreError: 97169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3587.559250] LustreError: 99864:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3587.563770] LustreError: 99864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3587.741863] Lustre: server umount lustre-MDT0001 complete [ 3599.906355] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3603.164687] Key type lgssc unregistered [ 3603.642938] LNet: 100397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3603.657800] LNetError: 100397:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3603.678838] LNet: Removed LNI 192.168.201.101@tcp [ 3604.822513] Key type .llcrypt unregistered [ 3604.832643] Key type ._llcrypt unregistered [ 3631.669512] Key type ._llcrypt registered [ 3631.673652] Key type .llcrypt registered [ 3631.854726] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3633.165782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.225088] alg: No test for adler32 (adler32-zlib) [ 3634.349567] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3634.609683] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3636.351823] Key type lgssc registered [ 3637.620820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3647.595293] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3647.658084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3663.328419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3668.511360] LustreError: 101620:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 3695.199278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.588731] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3707.468584] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3716.166825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.761632] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3716.773237] LustreError: 102532:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3716.784495] LustreError: 102532:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3716.793587] LustreError: 102532:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3716.962268] Lustre: server umount lustre-MDT0000 complete [ 3716.972822] LustreError: 102532:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3738.079566] Lustre: 100957:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769076736/real 1769076736] req@000000002e7c0940 x1855011316305152/t0(0) o400->MGC192.168.201.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769076752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3738.145287] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3738.164901] LustreError: 101633:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3758.559943] LustreError: 102963:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3758.702961] Lustre: server umount lustre-OST0000 complete [ 3770.688711] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 3781.429666] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3793.346893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3801.747389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3809.598798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3816.956422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3825.811166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3825.898505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.210890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3826.306477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3826.456631] Lustre: lustre-MDT0000: new disk, initializing [ 3826.723097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.770924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3832.040830] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3846.189321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3846.285934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3846.394576] Lustre: 105309:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3846.503299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3846.510384] Lustre: Skipped 1 previous similar message [ 3846.600537] Lustre: lustre-MDT0001: new disk, initializing [ 3846.757120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3846.821963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3846.846363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3851.952950] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3861.715751] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3865.940598] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3873.162869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.281731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3873.551529] Lustre: lustre-OST0000: new disk, initializing [ 3873.559956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3873.668238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.739667] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3883.612676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3883.625822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3883.762136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3887.726531] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3893.354381] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3893.619304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3898.756492] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3898.970840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3904.483589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3904.504248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.527768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3904.542347] Lustre: Skipped 1 previous similar message [ 3907.382207] LustreError: 108251:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3907.388962] LustreError: 108251:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3907.483506] Lustre: server umount lustre-OST0000 complete [ 3913.186263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3913.200190] LustreError: Skipped 1 previous similar message [ 3913.210398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.233128] Lustre: Skipped 1 previous similar message [ 3913.259711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.732873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.739945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.769054] Lustre: Skipped 1 previous similar message [ 3918.666907] LustreError: 108453:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3918.684837] LustreError: 108453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3918.919804] Lustre: server umount lustre-MDT0000 complete [ 3924.179199] LustreError: 105302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076938 with bad export cookie 12963612360955735743 [ 3924.206112] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3924.213196] LustreError: 105302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3924.366930] LustreError: 108654:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3924.375452] LustreError: 108654:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3924.683450] Lustre: server umount lustre-MDT0001 complete [ 3935.756896] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:15:48 (1769076948) [ 3939.957102] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3959.332096] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 3972.031946] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3984.849358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.733954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.959755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.218296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4017.197486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4017.388127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.929903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4017.978603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4018.105355] Lustre: lustre-MDT0000: new disk, initializing [ 4018.275964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.317047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4024.236891] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4037.560047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4037.653827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.793955] Lustre: 111792:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4037.961514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4037.976135] Lustre: Skipped 1 previous similar message [ 4038.128448] Lustre: lustre-MDT0001: new disk, initializing [ 4038.286072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4038.386759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4038.407847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4044.703791] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4054.764787] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4059.374740] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4068.600350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4068.731321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4069.198534] Lustre: lustre-OST0000: new disk, initializing [ 4069.201533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4069.400155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.450829] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4078.676863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4078.693712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4078.901099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4085.564983] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4091.141812] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4091.400827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4095.336121] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4095.632857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4099.561821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.576667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4099.602592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.623084] Lustre: Skipped 1 previous similar message [ 4103.753129] LustreError: 114734:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4103.755942] LustreError: 114734:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4104.060430] Lustre: server umount lustre-OST0000 complete [ 4109.793054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.799761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.805979] Lustre: Skipped 1 previous similar message [ 4109.820051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.835037] Lustre: Skipped 1 previous similar message [ 4114.265674] LustreError: 114936:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4114.286650] LustreError: 114936:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4114.695346] Lustre: server umount lustre-MDT0000 complete [ 4114.918801] LustreError: 111800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.944708] LustreError: 111800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4119.094734] LustreError: 111785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077133 with bad export cookie 12963612360955736625 [ 4119.121855] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.122535] LustreError: 111785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4119.766335] Lustre: server umount lustre-MDT0001 complete [ 4128.599464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.166315] LustreError: 115615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.259498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.312775] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4134.376585] LustreError: 115615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4138.466214] LustreError: 115615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.593470] LustreError: 115615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.596262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4149.153248] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4155.737244] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4161.190797] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4171.015074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4179.523292] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4186.106474] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4197.893759] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4204.517460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.519861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4204.531046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4204.540808] Lustre: Skipped 2 previous similar messages [ 4204.568279] LustreError: Skipped 1 previous similar message [ 4208.319164] LustreError: 117962:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4208.322108] LustreError: 117962:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4208.651531] Lustre: server umount lustre-OST0000 complete [ 4224.991874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.000649] Lustre: Skipped 5 previous similar messages [ 4227.039537] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4227.326302] Lustre: server umount lustre-MDT0000 complete [ 4230.114616] LustreError: 115614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.129114] LustreError: 115614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4232.343001] LustreError: 115595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077246 with bad export cookie 12963612360955737507 [ 4232.369169] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.873550] Lustre: server umount lustre-MDT0001 complete [ 4245.931983] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4249.662866] Key type lgssc unregistered [ 4250.039495] LNet: 118899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4250.047440] LNetError: 118899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4250.063622] LNet: Removed LNI 192.168.201.101@tcp [ 4251.043505] Key type .llcrypt unregistered [ 4251.054510] Key type ._llcrypt unregistered [ 4283.642829] Key type ._llcrypt registered [ 4283.652097] Key type .llcrypt registered [ 4283.841837] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 4302.297222] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4303.645265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4303.810134] alg: No test for adler32 (adler32-zlib) [ 4304.901776] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 4305.316705] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4307.079281] Key type lgssc registered [ 4308.912721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4318.923724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4326.391232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4333.466796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.104044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.655475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.853540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4351.902332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.264797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4353.282723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4353.327673] Lustre: lustre-MDT0000: new disk, initializing [ 4353.375729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.388053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4357.873277] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4369.276633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4369.401753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4369.478385] Lustre: 122030:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4369.535550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4369.543076] Lustre: Skipped 1 previous similar message [ 4369.646671] Lustre: lustre-MDT0001: new disk, initializing [ 4369.794687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4369.838408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4369.863887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4374.806963] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4383.782679] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4388.240836] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4396.297791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.400717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4396.683745] Lustre: lustre-OST0000: new disk, initializing [ 4396.688276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4396.751767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.530904] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4405.334948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4405.346838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4405.497406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4414.569786] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4420.862624] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4421.084904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4426.540087] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4426.818284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4431.336308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.345256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4431.354942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4431.377160] Lustre: Skipped 1 previous similar message [ 4435.023153] LustreError: 124976:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4435.302522] Lustre: server umount lustre-OST0000 complete [ 4441.568376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4441.579936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.584771] LustreError: Skipped 1 previous similar message [ 4441.588255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4441.646364] Lustre: Skipped 2 previous similar messages [ 4446.688886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.691436] Lustre: Skipped 1 previous similar message [ 4451.814800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.821648] Lustre: Skipped 1 previous similar message [ 4455.391264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4455.572219] LustreError: 125178:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4455.579428] LustreError: 125178:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4455.985546] Lustre: server umount lustre-MDT0000 complete [ 4460.844182] LustreError: 122023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077475 with bad export cookie 12353769073874528592 [ 4460.845451] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4460.849261] LustreError: 122023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4461.044085] LustreError: 125381:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4461.049978] LustreError: 125381:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4461.239367] Lustre: server umount lustre-MDT0001 complete [ 4471.394122] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:24:44 (1769077484) [ 4479.421310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.951691] LustreError: 126054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.964983] LustreError: 126054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4480.011438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4484.213723] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4485.103279] LustreError: 126054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.210588] LustreError: 126054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.529324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4493.890789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4498.850907] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4505.442923] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4509.382548] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4514.272872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4514.276258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.308100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.913062] LustreError: 127396:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4517.923023] LustreError: 127396:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4518.180383] Lustre: server umount lustre-MDT0000 complete [ 4519.395175] LustreError: 126054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.425515] LustreError: 126054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4523.237296] LustreError: 126033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077537 with bad export cookie 12353769073874529474 [ 4523.250752] LustreError: 126033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4523.258865] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4523.411693] LustreError: 127599:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4523.429627] LustreError: 127599:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4523.645604] Lustre: server umount lustre-MDT0001 complete [ 4532.784919] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:25:45 (1769077545) [ 4540.215699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4555.744478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4560.864624] LustreError: 128243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 4586.591373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4594.514894] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4601.526696] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4603.880458] LustreError: 128910:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4603.892163] LustreError: 128910:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4604.329733] Lustre: server umount lustre-OST0000 complete [ 4614.392559] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:27:06 (1769077626) [ 4622.190862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.952369] LustreError: 129582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.039910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.271601] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4632.549254] LustreError: 129582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.570025] LustreError: 129582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4639.650476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4646.753299] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4655.188742] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4659.203776] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4668.779487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4669.419230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4669.432334] Lustre: Skipped 1 previous similar message [ 4678.427696] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4687.762792] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4703.203823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.222765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4703.239806] Lustre: Skipped 3 previous similar messages [ 4703.243837] Lustre: Skipped 3 previous similar messages [ 4705.014925] LustreError: 131850:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4705.024122] LustreError: 131850:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4705.158475] Lustre: server umount lustre-MDT0000 complete [ 4708.324763] LustreError: 129576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.358785] LustreError: 129576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4710.447321] LustreError: 129563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077725 with bad export cookie 12353769073874530139 [ 4710.452772] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.460394] LustreError: 129563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4710.920835] Lustre: server umount lustre-MDT0001 complete [ 4726.693286] Lustre: server umount lustre-OST0000 complete [ 4737.101890] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:29:09 (1769077749) [ 4744.625564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.076905] LustreError: 132928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.155040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.529532] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4760.979690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4767.388560] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4773.544513] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4777.352569] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4786.084615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4789.836731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4796.723084] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4804.792818] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4811.807406] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4812.178736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4815.718442] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4815.939108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4820.458235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.459319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4820.474270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4820.489754] Lustre: Skipped 2 previous similar messages [ 4824.650295] LustreError: 135676:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4824.672201] LustreError: 135676:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4824.891772] Lustre: server umount lustre-OST0000 complete [ 4830.695901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.717042] Lustre: Skipped 2 previous similar messages [ 4835.809374] LustreError: 133837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.835529] LustreError: 133837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4837.393939] Lustre: server umount lustre-MDT0000 complete [ 4841.949019] LustreError: 132908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077856 with bad export cookie 12353769073874531210 [ 4841.957321] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4841.963982] LustreError: 132908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.309253] Lustre: server umount lustre-MDT0001 complete [ 4851.029531] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:31:04 (1769077864) [ 4858.190345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4873.762153] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4878.879766] LustreError: 136724:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 4904.608091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4904.617376] Lustre: Skipped 2 previous similar messages [ 4911.869715] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4917.430330] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4927.133632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.489119] LustreError: 120242:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000953797c5 x1855012018982016/t0(0) o250->MGC192.168.201.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4928.341478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4935.444254] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4945.981606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4950.971245] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4958.480156] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.078685] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4969.567037] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4969.775728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4975.839505] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4976.336096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4982.246248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.270302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4982.276743] Lustre: Skipped 2 previous similar messages [ 4982.303146] Lustre: Skipped 4 previous similar messages [ 4985.065838] LustreError: 139507:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4985.091737] LustreError: 139507:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4985.903048] Lustre: server umount lustre-MDT0000 complete [ 4987.365482] LustreError: 138634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4987.396925] LustreError: 138634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4990.517274] LustreError: 136732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078005 with bad export cookie 12353769073874532456 [ 4990.538500] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4990.826844] Lustre: server umount lustre-MDT0001 complete [ 5004.832904] Lustre: server umount lustre-OST0000 complete [ 5014.182387] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:33:47 (1769078027) [ 5022.180903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5037.665891] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5042.783522] LustreError: 140555:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 5075.755229] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5081.911900] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5091.250909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.931733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5097.556184] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5108.234432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5114.750730] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5119.933676] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5124.688793] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5133.184529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5133.315965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5133.446130] Lustre: lustre-OST0001: new disk, initializing [ 5133.448816] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5140.799157] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5142.624782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5142.646880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5142.751592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5148.862856] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5153.776409] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5154.037980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5158.907433] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5159.138848] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5163.488257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5163.505031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.517397] Lustre: Skipped 1 previous similar message [ 5163.523794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5163.535142] Lustre: Skipped 1 previous similar message [ 5167.496123] LustreError: 144290:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5167.507025] LustreError: 144290:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5167.634202] Lustre: server umount lustre-OST0000 complete [ 5173.748195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5178.931792] Lustre: server umount lustre-OST0001 complete [ 5190.041454] Lustre: server umount lustre-MDT0000 complete [ 5194.452872] LustreError: 141511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078209 with bad export cookie 12353769073874533401 [ 5194.463656] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.478563] LustreError: 141511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5195.073402] Lustre: server umount lustre-MDT0001 complete [ 5201.159550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5205.242844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5210.163343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5215.212157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5222.743111] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:37:15 (1769078235) [ 5224.901247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5227.808409] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5230.469887] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:37:22 (1769078242) [ 5241.273696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.720518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5241.782992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5242.082799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.094477] Lustre: Skipped 6 previous similar messages [ 5249.028136] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5260.072924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5260.211925] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5260.248310] Lustre: 146535:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5266.126384] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5273.486390] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5279.056281] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5289.139051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5289.508245] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5296.909147] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5299.445366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:193) [ 5305.345505] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5311.294696] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5311.823897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5318.794905] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5319.253735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5325.283098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5325.295986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.302946] LustreError: Skipped 2 previous similar messages [ 5325.303816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5325.303820] Lustre: Skipped 6 previous similar messages [ 5325.350282] Lustre: Skipped 5 previous similar messages [ 5327.932246] Lustre: server umount lustre-OST0000 complete [ 5350.897560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5359.332150] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5365.677628] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.694496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.706472] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5365.720939] Lustre: Skipped 1 previous similar message [ 5366.553466] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5371.659433] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5371.809711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5378.017691] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5378.309587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5383.808149] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff984946c19000.ost_server_uuid 50 [ 5385.643308] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff984946c19000.ost_server_uuid in FULL state after 0 sec [ 5385.866871] LustreError: 146543:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.1@tcp) returned error from blocking AST (req@00000000b415f7ff x1855012019120000 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c783d5b2/0xab716785057f856c lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.1@tcp remote: 0xaf706012233bd0eb expref: 5 pid: 146542 timeout: 5485 lvb_type: 0 [ 5385.927173] LustreError: lustre-MDT0000: A client on nid 192.168.201.1@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5385.950425] LustreError: 146531:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.1@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c783d5b2/0xab716785057f856c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.1@tcp remote: 0xaf706012233bd0eb expref: 6 pid: 146542 timeout: 0 lvb_type: 0 [ 5396.449642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5398.183517] Lustre: server umount lustre-OST0000 complete [ 5409.654835] Lustre: server umount lustre-MDT0000 complete [ 5411.828295] LustreError: 146542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5411.853921] LustreError: 146542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 5415.753431] LustreError: 146527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078430 with bad export cookie 12353769073874534577 [ 5415.754899] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5415.777125] LustreError: 146527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5430.239185] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5430.365580] LustreError: 151501:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5430.376303] LustreError: 151501:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 5430.817870] Lustre: server umount lustre-MDT0001 complete [ 5443.655756] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5446.643481] Key type lgssc unregistered [ 5446.874870] LNet: 152035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5446.883299] LNetError: 152035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5446.899682] LNet: Removed LNI 192.168.201.101@tcp [ 5447.701210] Key type .llcrypt unregistered [ 5447.707319] Key type ._llcrypt unregistered [ 5467.585264] Key type ._llcrypt registered [ 5467.586837] Key type .llcrypt registered [ 5467.672456] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5468.576930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5468.846831] alg: No test for adler32 (adler32-zlib) [ 5469.944164] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5470.218973] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5471.919422] Key type lgssc registered [ 5473.121197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5481.700318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5481.723401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.137185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.759019] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5492.706356] LustreError: 153409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.732457] LustreError: 153409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5497.827282] LustreError: 153409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.783836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5499.136539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5503.534571] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5509.305259] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5513.148616] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5521.404199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5521.955835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.116901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5533.395497] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5540.463585] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5547.493353] Lustre: Failing over lustre-MDT0000 [ 5547.651236] LustreError: 155658:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5547.732469] Lustre: server umount lustre-MDT0000 complete [ 5548.513455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.519913] Lustre: Skipped 1 previous similar message [ 5548.523952] LustreError: 153403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.535923] LustreError: 153403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5553.634109] LustreError: 153404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5553.654390] LustreError: 153404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5558.761357] LustreError: 154669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.773275] LustreError: 154669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5562.502196] LustreError: 153389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.1@tcp arrived at 1769078577 with bad export cookie 4936532902704139532 [ 5564.639825] Lustre: 152717:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769078563/real 1769078563] req@0000000064ad6c2f x1855013240467200/t0(0) o400->MGC192.168.201.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769078579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5564.669251] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5569.994079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5570.017317] LustreError: 154669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.030042] LustreError: 154669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5575.136949] LustreError: 152716:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee733251 x1855013240472832/t0(0) o250->MGC192.168.201.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5575.447865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.507321] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5580.324166] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5580.783865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5580.789040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5582.602270] Lustre: lustre-MDT0000: Denying connection for new client b39f46c6-9a39-45b8-874d-6472b42b1f54 (at 192.168.201.1@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5591.009854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.016950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.044728] Lustre: Skipped 1 previous similar message [ 5596.133334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.138331] Lustre: Skipped 2 previous similar messages [ 5601.260936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.274027] Lustre: Skipped 1 previous similar message [ 5605.343251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5605.384630] LustreError: 156593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5605.387805] LustreError: 156593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5605.617729] Lustre: server umount lustre-OST0000 complete [ 5609.209924] LustreError: 156797:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5609.216934] Lustre: 156159:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5609.224613] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5609.238983] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5609.248698] LustreError: 156159:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000351b45b0 x1855013240488192/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5609.266239] LustreError: 156159:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5609.280774] LustreError: 156159:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5609.287716] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5609.426075] LustreError: 156797:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5609.439275] LustreError: 156797:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5609.730791] Lustre: server umount lustre-MDT0000 complete [ 5611.494593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5611.526095] LustreError: 153404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.556135] LustreError: 153404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5612.794458] LustreError: 153390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078627 with bad export cookie 4936532902704139630 [ 5612.800329] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.804699] LustreError: 153390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5612.807913] LustreError: 152716:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5619.017613] LustreError: 156998:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5619.035164] LustreError: 156998:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5619.360091] Lustre: server umount lustre-MDT0001 complete [ 5631.572790] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5634.494431] Key type lgssc unregistered [ 5634.823102] LNet: 157534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5634.830593] LNetError: 157534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5634.852571] LNet: Removed LNI 192.168.201.101@tcp [ 5635.692771] Key type .llcrypt unregistered [ 5635.694801] Key type ._llcrypt unregistered [ 5658.275784] Key type ._llcrypt registered [ 5658.277412] Key type .llcrypt registered [ 5658.377600] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5659.056364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5659.182950] alg: No test for adler32 (adler32-zlib) [ 5660.220246] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5660.434121] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5662.151209] Key type lgssc registered [ 5663.219540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5675.106276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5675.128685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.710635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.764895] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5687.269329] LustreError: 158933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.295940] LustreError: 158933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5689.703312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5690.129835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.853483] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5699.372864] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.281817] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5710.924660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5711.308666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.731041] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5721.514817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 5724.195858] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5730.787452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.803702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.832857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.810609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5731.816907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.820121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.832692] Lustre: Skipped 1 previous similar message [ 5736.600022] LustreError: 161178:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5736.731528] Lustre: server umount lustre-OST0000 complete [ 5741.540360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5741.552657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.572959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5746.439360] LustreError: 161382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5746.458114] LustreError: 161382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5746.716902] Lustre: server umount lustre-MDT0000 complete [ 5747.171344] LustreError: 158929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.184262] LustreError: 158929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5751.595181] LustreError: 158914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078766 with bad export cookie 1222282256069499195 [ 5751.602334] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5751.605880] LustreError: 158914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5751.779758] LustreError: 161583:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5751.797551] LustreError: 161583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5752.005412] Lustre: server umount lustre-MDT0001 complete [ 5766.276982] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5770.653090] Key type lgssc unregistered [ 5771.112664] LNet: 162116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5771.117927] LNetError: 162116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5772.202741] LNet: Removed LNI 192.168.201.101@tcp [ 5773.728158] Key type .llcrypt unregistered [ 5773.731491] Key type ._llcrypt unregistered [ 5808.407225] Key type ._llcrypt registered [ 5808.410570] Key type .llcrypt registered [ 5808.523037] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5809.521436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5809.562117] alg: No test for adler32 (adler32-zlib) [ 5810.860653] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5811.048352] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5812.703949] Key type lgssc registered [ 5813.979855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5821.800616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5821.816631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.276291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.925743] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5832.673813] LustreError: 163798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.680735] LustreError: 163798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5837.678876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5837.797053] LustreError: 163798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.991991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5843.935381] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5851.224312] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5855.300694] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5864.314271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5864.972504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.030370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 5872.101425] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5878.522161] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5892.069812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5892.073367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.087934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.600564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.603496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5893.604856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.621208] Lustre: Skipped 1 previous similar message [ 5897.478833] LustreError: 166078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5897.689583] Lustre: server umount lustre-OST0000 complete [ 5903.843216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.875859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.887620] Lustre: Skipped 1 previous similar message [ 5908.681297] LustreError: 166280:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5908.694114] LustreError: 166280:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5908.882211] Lustre: server umount lustre-MDT0000 complete [ 5908.984657] LustreError: 165968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.012766] LustreError: 165968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5913.312596] LustreError: 163778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078927 with bad export cookie 1666849387262818856 [ 5913.344268] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.514904] LustreError: 166482:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5913.520300] LustreError: 166482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5913.698931] Lustre: server umount lustre-MDT0001 complete [ 5925.075476] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5928.868778] Key type lgssc unregistered [ 5929.291446] LNet: 167014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5929.306613] LNetError: 167014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5930.347403] LNet: Removed LNI 192.168.201.101@tcp [ 5931.318477] Key type .llcrypt unregistered [ 5931.325731] Key type ._llcrypt unregistered [ 5960.507979] Key type ._llcrypt registered [ 5960.518522] Key type .llcrypt registered [ 5960.702327] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5962.426416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5962.599800] alg: No test for adler32 (adler32-zlib) [ 5963.736394] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5964.043116] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5965.743280] Key type lgssc registered [ 5967.341954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5980.176367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5980.211269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.800864] Lustre: *** cfs_fail_loc=135, val=0*** [ 5981.974540] LustreError: 168521:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 5981.987504] LustreError: 168521:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -2 [ 5982.011154] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5982.011154] [ 5982.026490] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5982.049180] LustreError: 168481:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5982.070869] LustreError: 168481:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5982.087924] LustreError: 168481:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 5982.101142] LustreError: 168481:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 5982.118346] LustreError: 168481:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5982.255354] Lustre: server umount lustre-MDT0000 complete [ 5982.260463] LustreError: 168481:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5991.012090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.687021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.304705] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6002.176569] LustreError: 168895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.221794] LustreError: 168895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6007.265297] LustreError: 168895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.997085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6008.522312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6014.938096] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6022.719676] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6028.343834] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6032.546685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.551172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.557652] Lustre: Skipped 1 previous similar message [ 6032.566589] Lustre: Skipped 1 previous similar message [ 6037.792417] LustreError: 170240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6037.796034] LustreError: 170240:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6037.985305] LustreError: 168894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.994347] LustreError: 168894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6038.005108] Lustre: server umount lustre-MDT0000 complete [ 6042.867830] LustreError: 168877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079057 with bad export cookie 17575155887091426471 [ 6042.905959] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6043.071833] LustreError: 170443:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6043.076936] LustreError: 170443:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6043.359897] Lustre: server umount lustre-MDT0001 complete [ 6055.778754] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6058.970191] Key type lgssc unregistered [ 6059.269664] LNet: 170975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6059.280160] LNetError: 170975:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6060.329556] LNet: Removed LNI 192.168.201.101@tcp [ 6061.253917] Key type .llcrypt unregistered [ 6061.255985] Key type ._llcrypt unregistered [ 6085.689762] Key type ._llcrypt registered [ 6085.692910] Key type .llcrypt registered [ 6085.885920] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6086.221706] Key type .llcrypt unregistered [ 6086.232496] Key type ._llcrypt unregistered [ 6109.461942] Key type ._llcrypt registered [ 6109.463799] Key type .llcrypt registered [ 6109.567974] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 6110.945057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.101265] alg: No test for adler32 (adler32-zlib) [ 6112.442264] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6112.733662] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 6114.567291] Key type lgssc registered [ 6116.531313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6128.137670] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6128.166580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6143.906115] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6149.023230] LustreError: 173072:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 6175.711242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.573425] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6187.068709] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6195.517477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.060218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.136187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 6200.467040] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6201.322621] LustreError: 173098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6201.347862] LustreError: 173098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6206.438435] LustreError: 173098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.185827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6210.548301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.093489] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6223.344238] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6227.785702] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6230.568036] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6246.381303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.394638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.405692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.405507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.414868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.424722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.442241] Lustre: Skipped 1 previous similar message [ 6250.041182] LustreError: 175798:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6250.197863] Lustre: server umount lustre-OST0000 complete [ 6256.610210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6256.617191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.662212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.985852] LustreError: 176000:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6260.996398] LustreError: 176000:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6261.167814] Lustre: server umount lustre-MDT0000 complete [ 6262.763646] LustreError: 174001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.789488] LustreError: 174001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6265.293273] LustreError: 173081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079279 with bad export cookie 9209654965362078647 [ 6265.299129] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6265.314947] LustreError: 173081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6265.509962] LustreError: 176201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6265.516687] LustreError: 176201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6265.786090] Lustre: server umount lustre-MDT0001 complete [ 6278.503756] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6281.848787] Key type lgssc unregistered [ 6282.375934] LNet: 176733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6282.389380] LNetError: 176733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6282.408204] LNet: Removed LNI 192.168.201.101@tcp [ 6283.518420] Key type .llcrypt unregistered [ 6283.520111] Key type ._llcrypt unregistered [ 6309.735328] Key type ._llcrypt registered [ 6309.737839] Key type .llcrypt registered [ 6309.857254] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 6311.165594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6311.270407] alg: No test for adler32 (adler32-zlib) [ 6312.431764] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6312.649311] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 6314.329808] Key type lgssc registered [ 6315.713923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6324.831847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6324.853215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.617301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.468685] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6336.992494] LustreError: 178131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.026018] LustreError: 178131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6342.118903] LustreError: 178131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.480681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6343.009515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6348.090551] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6356.398925] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6361.557565] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6370.753426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6371.328370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.410943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:353) [ 6377.597336] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6382.872498] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6387.537883] Lustre: Failing over lustre-MDT0000 [ 6387.650295] LustreError: 180428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6387.682331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.699518] LustreError: 178125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.707865] LustreError: 178125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6387.749101] Lustre: server umount lustre-MDT0000 complete [ 6389.841818] LustreError: 178127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6389.864726] LustreError: 178127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6394.936952] LustreError: 178127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.973804] LustreError: 178127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6404.068404] LustreError: 178130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6404.105857] LustreError: 178130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6405.089936] Lustre: 177441:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079403/real 1769079403] req@00000000e1e7bf60 x1855014124413184/t0(0) o400->MGC192.168.201.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6405.147696] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6406.990270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.364827] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xed205f3551fe2eaf [ 6415.387145] Lustre: MGC192.168.201.101@tcp: Connection restored to 0@lo (at 0@lo) [ 6415.424552] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (not set up) [ 6415.690669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.773742] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6420.533964] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6420.983355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6420.995129] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6420.995917] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6421.065352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:385) [ 6425.235728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6434.446611] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6448.115718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6448.120717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.129358] Lustre: Skipped 2 previous similar messages [ 6448.136269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6449.345741] LustreError: 181789:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6449.355289] LustreError: 181789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6449.653159] Lustre: server umount lustre-OST0000 complete [ 6456.799971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6456.800761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.840378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6459.959622] LustreError: 181990:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6459.963365] LustreError: 181990:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6460.125421] Lustre: server umount lustre-MDT0000 complete [ 6461.935913] LustreError: 178131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.962594] LustreError: 178131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6463.712207] LustreError: 178109:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079478 with bad export cookie 17086761668857179823 [ 6463.721752] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.728208] LustreError: 178109:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6474.244715] LustreError: 182192:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6474.249431] LustreError: 182192:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6474.396270] Lustre: server umount lustre-MDT0001 complete [ 6484.125487] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6486.655200] Key type lgssc unregistered [ 6487.083703] LNet: 182725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6487.095278] LNetError: 182725:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6487.120409] LNet: Removed LNI 192.168.201.101@tcp [ 6488.036838] Key type .llcrypt unregistered [ 6488.038763] Key type ._llcrypt unregistered [ 6511.248427] Key type ._llcrypt registered [ 6511.253726] Key type .llcrypt registered [ 6511.364054] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 6512.445362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6512.700646] alg: No test for adler32 (adler32-zlib) [ 6513.807159] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6513.988062] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 6515.639208] Key type lgssc registered [ 6516.621805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6523.428634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6523.445059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.017858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6529.075567] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6535.139210] LustreError: 184120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.160669] LustreError: 184120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6537.311435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6537.716081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6542.287973] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6547.450756] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6550.795079] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.725341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6555.969758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6561.362423] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6566.017042] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6570.360776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:417) [ 6573.042960] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6575.350156] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6588.550275] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6599.138026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6599.144045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.174072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6601.184466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6601.196412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6601.199192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.216162] Lustre: Skipped 1 previous similar message [ 6604.742819] LustreError: 186557:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6604.993065] Lustre: server umount lustre-OST0000 complete [ 6609.376089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6609.383045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.402291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.427572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.430152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6614.883045] LustreError: 186758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6614.888380] LustreError: 186758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6616.544883] LustreError: 184116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.555103] LustreError: 184116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6617.012945] LustreError: 186758:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6617.024054] LustreError: 186758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6617.068525] Lustre: server umount lustre-MDT0000 complete [ 6621.068616] LustreError: 184101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079635 with bad export cookie 9945697484910095085 [ 6621.077501] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.082989] LustreError: 184101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6621.215336] LustreError: 186961:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6621.226124] LustreError: 186961:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6621.418200] Lustre: server umount lustre-MDT0001 complete [ 6632.677437] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6635.761933] Key type lgssc unregistered [ 6636.041925] LNet: 187493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6636.048544] LNetError: 187493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6636.061894] LNet: Removed LNI 192.168.201.101@tcp [ 6636.933577] Key type .llcrypt unregistered [ 6636.935222] Key type ._llcrypt unregistered [ 6661.099581] Key type ._llcrypt registered [ 6661.102496] Key type .llcrypt registered [ 6661.283565] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 6662.859540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6662.912566] alg: No test for adler32 (adler32-zlib) [ 6663.941667] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6664.176807] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 6665.863355] Key type lgssc registered [ 6667.243143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6674.792342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6674.816892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6676.297876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6680.656535] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6685.684542] LustreError: 188897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.723076] LustreError: 188897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6688.882183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6689.274510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6693.270622] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6698.783389] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6702.566271] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6709.086695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6709.414138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.508500] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6720.580362] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6720.694110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:449) [ 6726.498082] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6739.091803] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6746.659531] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6752.577744] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6761.965144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.965947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6761.984702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.133727] LustreError: 192493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6766.259654] Lustre: server umount lustre-OST0000 complete [ 6771.172741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.184683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.228862] Lustre: Skipped 1 previous similar message [ 6771.232626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6771.242323] Lustre: Skipped 1 previous similar message [ 6772.193453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.180820] LustreError: 192694:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6776.184064] LustreError: 192694:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6776.420962] Lustre: server umount lustre-MDT0000 complete [ 6777.311941] LustreError: 188898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.325861] LustreError: 188898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6780.179570] LustreError: 188877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079794 with bad export cookie 31567161027537522 [ 6780.186990] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6780.198376] LustreError: 188877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6780.374769] LustreError: 192896:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6780.391117] LustreError: 192896:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6780.694041] Lustre: server umount lustre-MDT0001 complete [ 6791.762435] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6794.548410] Key type lgssc unregistered [ 6794.849950] LNet: 193428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6794.855925] LNetError: 193428:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6795.885199] LNet: Removed LNI 192.168.201.101@tcp [ 6796.594256] Key type .llcrypt unregistered [ 6796.596220] Key type ._llcrypt unregistered [ 6819.984685] Key type ._llcrypt registered [ 6819.986859] Key type .llcrypt registered [ 6820.060343] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 6821.389870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6821.459070] alg: No test for adler32 (adler32-zlib) [ 6822.496422] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6822.778165] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 6824.471202] Key type lgssc registered [ 6825.720350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6833.728950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6833.747822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.295769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.962869] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6844.393693] LustreError: 194822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.414132] LustreError: 194822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6848.731891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6849.010383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6852.543467] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6857.912199] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6861.545774] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6868.151581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6868.511919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6873.771846] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6878.911160] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6879.799226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:481) [ 6894.048646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6894.059954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.073306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.077116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6895.085478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.089465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.102937] Lustre: Skipped 1 previous similar message [ 6899.529160] LustreError: 197218:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6899.722548] Lustre: server umount lustre-OST0000 complete [ 6905.312655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6905.312835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.320473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.332582] Lustre: Skipped 1 previous similar message [ 6910.437498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6910.445880] Lustre: Skipped 2 previous similar messages [ 6911.302062] LustreError: 197419:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6911.306258] LustreError: 197419:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6911.351941] Lustre: server umount lustre-MDT0000 complete [ 6914.583529] LustreError: 194800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079929 with bad export cookie 4912310276520657951 [ 6914.589638] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6914.591892] LustreError: 194800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6914.739471] LustreError: 197621:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6914.746703] LustreError: 197621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6914.913866] Lustre: server umount lustre-MDT0001 complete [ 6924.110298] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 6926.553866] Key type lgssc unregistered [ 6926.800672] LNet: 198152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6926.806578] LNetError: 198152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6927.854824] LNet: Removed LNI 192.168.201.101@tcp [ 6928.803371] Key type .llcrypt unregistered [ 6928.807891] Key type ._llcrypt unregistered [ 6945.377704] Key type ._llcrypt registered [ 6945.379819] Key type .llcrypt registered [ 6945.486312] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 6946.690821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6946.718663] alg: No test for adler32 (adler32-zlib) [ 6947.803916] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 6947.998511] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 6949.639692] Key type lgssc registered [ 6951.021995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6957.879106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6957.895597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6959.255127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.171266] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6968.801947] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.833653] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6970.534979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6970.815990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6974.232551] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6979.499112] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6982.950968] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6988.311595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6988.669312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.415481] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 6999.306452] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7004.019052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:513) [ 7019.490136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.494651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7019.497751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.508439] Lustre: Skipped 1 previous similar message [ 7023.486355] LustreError: 201800:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7023.680548] Lustre: server umount lustre-OST0000 complete [ 7027.174818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7027.183261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.192687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7027.197722] Lustre: Skipped 1 previous similar message [ 7029.729423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.732763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.755840] Lustre: Skipped 1 previous similar message [ 7033.191447] LustreError: 202001:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7033.195822] LustreError: 202001:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7033.308797] Lustre: server umount lustre-MDT0000 complete [ 7034.853967] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.873896] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7036.108473] LustreError: 199362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080050 with bad export cookie 1383587314943659364 [ 7036.121539] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.233178] LustreError: 202203:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7036.240413] LustreError: 202203:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7036.373769] Lustre: server umount lustre-MDT0001 complete [ 7044.491818] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 7046.487394] Key type lgssc unregistered [ 7046.728651] LNet: 202734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7046.733443] LNetError: 202734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7046.758173] LNet: Removed LNI 192.168.201.101@tcp [ 7047.341649] Key type .llcrypt unregistered [ 7047.344580] Key type ._llcrypt unregistered [ 7062.360527] Key type ._llcrypt registered [ 7062.362673] Key type .llcrypt registered [ 7062.449691] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7063.215836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7063.237229] alg: No test for adler32 (adler32-zlib) [ 7064.255228] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7064.433540] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 7066.066281] Key type lgssc registered [ 7067.108244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7073.436356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7073.452444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.897566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.314878] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7084.862826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7085.025888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7085.180705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.428901] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7092.728183] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7096.365499] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7102.513274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7102.861697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.524640] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7113.064867] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7120.326839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:545) [ 7132.090194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7132.154862] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7132.309407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7136.492800] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7141.497178] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7142.814179] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7154.329645] Lustre: Permanently deactivating lustre-OST0001 [ 7154.348834] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7164.449633] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7170.251357] Lustre: Permanently reactivating lustre-OST0001 [ 7170.261718] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7170.265163] Lustre: Skipped 2 previous similar messages [ 7171.764911] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.772800] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7171.777753] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7171.788793] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7171.811810] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7180.860300] LustreError: 207461:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7180.972742] Lustre: server umount lustre-OST0001 complete [ 7182.311992] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7182.316107] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7182.320886] LustreError: Skipped 1 previous similar message [ 7182.322321] LustreError: 205562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.322329] LustreError: 205562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7182.370365] Lustre: Skipped 2 previous similar messages [ 7186.913651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7186.914732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.919041] LustreError: Skipped 1 previous similar message [ 7186.928988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7186.935040] Lustre: Skipped 1 previous similar message [ 7186.958330] Lustre: Skipped 1 previous similar message [ 7187.949090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7187.956343] Lustre: Skipped 1 previous similar message [ 7187.956772] LustreError: 205560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.977119] LustreError: 205560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7190.083128] LustreError: 207661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7190.086139] LustreError: 207661:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7190.234745] Lustre: server umount lustre-OST0000 complete [ 7197.666739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7197.673097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.685147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7199.771654] LustreError: 207864:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7199.780073] LustreError: 207864:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7199.937620] Lustre: server umount lustre-MDT0000 complete [ 7203.172683] LustreError: 203945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080217 with bad export cookie 7513668619987862744 [ 7203.179347] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.190069] LustreError: 203945:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7203.374202] LustreError: 208065:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7203.381439] LustreError: 208065:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7203.660556] Lustre: server umount lustre-MDT0001 complete [ 7212.353133] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 7214.713901] Key type lgssc unregistered [ 7214.889664] LNet: 208597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7214.895326] LNetError: 208597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7214.907841] LNet: Removed LNI 192.168.201.101@tcp [ 7215.550376] Key type .llcrypt unregistered [ 7215.552961] Key type ._llcrypt unregistered [ 7242.835759] Key type ._llcrypt registered [ 7242.838448] Key type .llcrypt registered [ 7242.959430] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 7254.993176] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7255.627672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7255.731899] alg: No test for adler32 (adler32-zlib) [ 7256.747996] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7256.890312] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 7258.520076] Key type lgssc registered [ 7259.339666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7265.983875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7271.030792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7276.033360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7281.786435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7287.196827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7287.241948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7287.252042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.382912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7288.401455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7288.458970] Lustre: lustre-MDT0000: new disk, initializing [ 7288.526634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.548296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7291.513697] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7300.353096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7300.411458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7300.481241] Lustre: 212523:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7300.524705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7300.527514] Lustre: Skipped 1 previous similar message [ 7300.581976] Lustre: lustre-MDT0001: new disk, initializing [ 7300.662279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7300.700269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7300.714924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7304.479746] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7310.805704] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7314.056308] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7320.403658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.485643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7320.835902] Lustre: lustre-OST0000: new disk, initializing [ 7320.843075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7320.961791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7326.378395] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7330.388841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7330.395290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7330.535657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7333.065667] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7337.297550] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7337.471794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7341.021645] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7341.196184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7346.146365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7346.149376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.160666] LustreError: Skipped 1 previous similar message [ 7346.186682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7346.200041] Lustre: Skipped 1 previous similar message [ 7348.796231] LustreError: 215459:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7348.871513] Lustre: server umount lustre-OST0000 complete [ 7356.387789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.389842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7356.404243] Lustre: Skipped 2 previous similar messages [ 7358.179517] LustreError: 215662:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7358.183747] LustreError: 215662:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7358.275573] Lustre: server umount lustre-MDT0000 complete [ 7361.502810] LustreError: 212513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080376 with bad export cookie 12083157922865445842 [ 7361.504696] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7361.521165] LustreError: 212513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7361.677702] LustreError: 215863:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7361.682110] LustreError: 215863:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7362.150758] Lustre: server umount lustre-MDT0001 complete [ 7369.231397] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 06:13:02 (1769080382) [ 7375.006569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.488494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.731940] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7380.968180] LustreError: 216533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.985430] LustreError: 216533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7386.083830] LustreError: 216533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.457132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7386.827433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7390.623260] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7395.297355] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7398.206838] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7404.244911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7404.504246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.386561] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7414.084900] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7426.479543] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7432.717181] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7442.144156] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7449.746826] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7459.528763] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7476.657333] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7476.661854] Lustre: Skipped 1 previous similar message [ 7516.039945] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7516.044804] Lustre: Skipped 4 previous similar messages [ 7584.079833] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7584.087099] Lustre: Skipped 8 previous similar messages [ 7598.321603] LustreError: 220011:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7598.325667] LustreError: 220011:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7598.439088] Lustre: server umount lustre-OST0000 complete [ 7602.144047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7602.151728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7602.162408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7602.166497] Lustre: Skipped 1 previous similar message [ 7605.218085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.226218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.337109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7610.342988] Lustre: Skipped 2 previous similar messages [ 7611.370776] LustreError: 220211:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7611.377614] LustreError: 220211:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7611.421297] Lustre: server umount lustre-MDT0000 complete [ 7613.689957] LustreError: 219411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080628 with bad export cookie 12083157922865446724 [ 7613.695514] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.700470] LustreError: 219411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7624.169507] LustreError: 220413:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7624.176512] LustreError: 220413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7624.275525] Lustre: server umount lustre-MDT0001 complete [ 7630.728229] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 7632.495286] Key type lgssc unregistered [ 7632.669643] LNet: 220946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7632.676241] LNetError: 220946:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7632.690730] LNet: Removed LNI 192.168.201.101@tcp [ 7633.136219] Key type .llcrypt unregistered [ 7633.138242] Key type ._llcrypt unregistered [ 7647.520655] Key type ._llcrypt registered [ 7647.522257] Key type .llcrypt registered [ 7647.604666] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7648.254407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7648.337282] alg: No test for adler32 (adler32-zlib) [ 7649.311923] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7649.451634] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 7651.055179] Key type lgssc registered [ 7651.644881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7656.231150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7656.245454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7657.600585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.981969] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7665.236090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7665.495949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7668.218987] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7672.151610] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7674.588832] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7679.266095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7679.499811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.692510] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7687.240535] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7697.888895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7697.894965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.906492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7700.960135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7700.963845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7700.969420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.986539] Lustre: Skipped 1 previous similar message [ 7703.428049] LustreError: 224712:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7703.553933] Lustre: server umount lustre-OST0000 complete [ 7706.082756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.093974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7706.099045] Lustre: Skipped 1 previous similar message [ 7711.200319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7711.207530] Lustre: Skipped 1 previous similar message [ 7712.173659] LustreError: 224917:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7712.183430] LustreError: 224917:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7712.260915] Lustre: server umount lustre-MDT0000 complete [ 7714.652995] LustreError: 222296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080729 with bad export cookie 2565623180337764263 [ 7714.663728] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7714.746942] LustreError: 225118:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7714.753978] LustreError: 225118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7714.892375] Lustre: server umount lustre-MDT0001 complete [ 7721.480835] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 7723.256670] Key type lgssc unregistered [ 7723.460621] LNet: 225650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7723.463462] LNetError: 225650:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7723.474217] LNet: Removed LNI 192.168.201.101@tcp [ 7723.952174] Key type .llcrypt unregistered [ 7723.956271] Key type ._llcrypt unregistered [ 7737.899533] Key type ._llcrypt registered [ 7737.901542] Key type .llcrypt registered [ 7737.987501] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7738.750852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7738.809639] alg: No test for adler32 (adler32-zlib) [ 7739.796056] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7739.968547] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 7741.575190] Key type lgssc registered [ 7742.330704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8009.306504] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 8011.089029] Key type lgssc unregistered [ 8011.288805] LNet: 227614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8011.292829] LNetError: 227614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8011.309354] LNet: Removed LNI 192.168.201.101@tcp [ 8011.849580] Key type .llcrypt unregistered [ 8011.851284] Key type ._llcrypt unregistered [ 8027.744620] Key type ._llcrypt registered [ 8027.746632] Key type .llcrypt registered [ 8027.807709] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 8037.660980] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 8038.537337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8038.569598] alg: No test for adler32 (adler32-zlib) [ 8039.672652] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8039.909174] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 8041.607221] Key type lgssc registered [ 8043.062826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8056.639036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8065.325543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8075.494700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8087.396812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8089.765836] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7755 sec ========== 06:25:02 (1769081102) [ 8091.885155] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:25:04 (1769081104) === [ 8095.578061] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:25:08 (1769081108) === [ 8132.384334] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 8134.877277] Key type lgssc unregistered [ 8135.129951] LNet: 232784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8135.144633] LNetError: 232784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8135.162560] LNet: Removed LNI 192.168.201.101@tcp [ 8135.891801] Key type .llcrypt unregistered [ 8135.893870] Key type ._llcrypt unregistered