[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 535113212 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003184] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008649] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.022827] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026797] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] *** VALIDATE tmpfs *** [ 0.031425] *** VALIDATE proc *** [ 0.032268] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037034] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041534] debug: unmapping init [mem 0xffffffffa0059000-0xffffffffa0060fff] [ 0.043178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052291] rcu: Hierarchical SRCU implementation. [ 0.054496] smp: Bringing up secondary CPUs ... [ 0.055595] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059087] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.102588] node 0 deferred pages initialised in 38ms [ 0.105031] devtmpfs: initialized [ 0.105779] x86/mm: Memory block size: 128MB [ 0.108375] gcov: version magic: 0x41383552 [ 0.110211] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113260] pinctrl core: initialized pinctrl subsystem [ 0.115136] [ 0.115447] ************************************************************* [ 0.117009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.119014] ** ** [ 0.122076] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.124016] ** ** [ 0.127018] ** This means that this kernel is built to expose internal ** [ 0.130081] ** IOMMU data structures, which may compromise security on ** [ 0.133017] ** your system. ** [ 0.136065] ** ** [ 0.139018] ** If you see this message and you are not debugging the ** [ 0.141012] ** kernel, report this immediately to your vendor! ** [ 0.144013] ** ** [ 0.147014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150073] ************************************************************* [ 0.153953] NET: Registered protocol family 16 [ 0.155539] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165849] cpuidle: using governor menu [ 0.167914] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170555] PCI: Using configuration type 1 for base access [ 0.173145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187090] cryptd: max_cpu_qlen set to 1000 [ 0.190298] ACPI: Added _OSI(Module Device) [ 0.192025] ACPI: Added _OSI(Processor Device) [ 0.193011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199964] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205032] ACPI: Interpreter enabled [ 0.206036] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207010] ACPI: Using IOAPIC for interrupt routing [ 0.209131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212322] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222693] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.224039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236548] acpiphp: Slot [2] registered [ 0.238191] acpiphp: Slot [5] registered [ 0.240104] acpiphp: Slot [6] registered [ 0.241143] acpiphp: Slot [7] registered [ 0.243121] acpiphp: Slot [8] registered [ 0.245164] acpiphp: Slot [9] registered [ 0.246109] acpiphp: Slot [10] registered [ 0.248122] acpiphp: Slot [3] registered [ 0.250131] acpiphp: Slot [4] registered [ 0.251086] acpiphp: Slot [11] registered [ 0.253105] acpiphp: Slot [12] registered [ 0.255098] acpiphp: Slot [13] registered [ 0.257120] acpiphp: Slot [14] registered [ 0.258085] acpiphp: Slot [15] registered [ 0.259096] acpiphp: Slot [16] registered [ 0.261185] acpiphp: Slot [17] registered [ 0.262086] acpiphp: Slot [18] registered [ 0.264096] acpiphp: Slot [19] registered [ 0.265098] acpiphp: Slot [20] registered [ 0.267106] acpiphp: Slot [21] registered [ 0.268118] acpiphp: Slot [22] registered [ 0.270159] acpiphp: Slot [23] registered [ 0.272092] acpiphp: Slot [24] registered [ 0.273095] acpiphp: Slot [25] registered [ 0.275112] acpiphp: Slot [26] registered [ 0.277116] acpiphp: Slot [27] registered [ 0.278091] acpiphp: Slot [28] registered [ 0.280192] acpiphp: Slot [29] registered [ 0.282092] acpiphp: Slot [30] registered [ 0.283109] acpiphp: Slot [31] registered [ 0.285058] PCI host bridge to bus 0000:00 [ 0.286018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.288024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.291080] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.296023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304085] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.308000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317849] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.330026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334143] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.335804] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.337035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339853] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.360019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.364024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369349] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413243] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.452000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.462016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.475013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.480865] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.490019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.503021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.511000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.515014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.519018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.530019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.541112] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.547018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.554017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.569019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.578773] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.580363] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.583412] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.585336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.587152] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.593186] iommu: Default domain type: Passthrough [ 0.594426] SCSI subsystem initialized [ 0.596103] ACPI: bus type USB registered [ 0.597060] usbcore: registered new interface driver usbfs [ 0.598000] usbcore: registered new interface driver hub [ 0.598000] usbcore: registered new device driver usb [ 0.599110] pps_core: LinuxPPS API ver. 1 registered [ 0.600005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.602045] PTP clock support registered [ 0.603186] EDAC MC: Ver: 3.0.0 [ 0.605054] PCI: Using ACPI for IRQ routing [ 0.606621] NetLabel: Initializing [ 0.607006] NetLabel: domain hash size = 128 [ 0.607922] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.609058] NetLabel: unlabeled traffic allowed by default [ 0.610244] vgaarb: loaded [ 0.611269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.613006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.618000] clocksource: Switched to clocksource kvm-clock [ 0.710576] VFS: Disk quotas dquot_6.6.0 [ 0.712350] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.714082] *** VALIDATE ramfs *** [ 0.714746] *** VALIDATE hugetlbfs *** [ 0.715776] pnp: PnP ACPI init [ 0.719224] pnp: PnP ACPI: found 6 devices [ 0.735652] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.738151] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.739423] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.740570] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.742116] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.743756] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.746279] NET: Registered protocol family 2 [ 0.748694] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.753066] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.755188] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.758726] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.761340] TCP: Hash tables configured (established 65536 bind 65536) [ 0.764251] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.766983] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.769283] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.772152] NET: Registered protocol family 1 [ 0.774544] RPC: Registered named UNIX socket transport module. [ 0.776936] RPC: Registered udp transport module. [ 0.778308] RPC: Registered tcp transport module. [ 0.779358] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.780937] NET: Registered protocol family 44 [ 0.782411] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.784841] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.787147] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.789385] PCI: CLS 0 bytes, default 64 [ 0.791962] Unpacking initramfs... [ 2.190940] debug: unmapping init [mem 0xffff8cd37cc54000-0xffff8cd37ffbffff] [ 2.195096] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.197294] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.199645] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.702143] Initialise system trusted keyrings [ 2.703959] Key type blacklist registered [ 2.705884] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.716094] zbud: loaded [ 2.719332] *** VALIDATE nfs *** [ 2.720788] *** VALIDATE nfs4 *** [ 2.722633] pstore: using deflate compression [ 2.725827] Platform Keyring initialized [ 2.826457] NET: Registered protocol family 38 [ 2.828455] Key type asymmetric registered [ 2.830229] Asymmetric key parser 'x509' registered [ 2.832367] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.835940] io scheduler mq-deadline registered [ 2.837953] io scheduler kyber registered [ 2.839980] io scheduler bfq registered [ 2.842112] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.845453] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.848954] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.851249] ACPI: Power Button [PWRF] [ 2.856597] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.861717] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.872313] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.879522] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.892741] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.920250] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.949195] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.953627] Non-volatile memory driver v1.3 [ 2.955354] Linux agpgart interface v0.103 [ 2.985840] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 2.989401] vda: detected capacity change from 0 to 74731520 [ 3.007535] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.010735] vdb: detected capacity change from 0 to 1073741824 [ 3.037102] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.040447] vdc: detected capacity change from 0 to 2621440000 [ 3.056052] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.059212] vdd: detected capacity change from 0 to 2621440000 [ 3.073594] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.076964] vde: detected capacity change from 0 to 4294967296 [ 3.097718] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.100989] vdf: detected capacity change from 0 to 4294967296 [ 3.112348] libphy: Fixed MDIO Bus: probed [ 3.117956] usbcore: registered new interface driver usbserial_generic [ 3.121031] usbserial: USB Serial support registered for generic [ 3.124077] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.128868] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.131119] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.133746] mousedev: PS/2 mouse device common for all mice [ 3.137323] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.139549] rtc_cmos 00:05: RTC can wake from S4 [ 3.145159] rtc_cmos 00:05: registered as rtc0 [ 3.147091] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.150027] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.151926] intel_pstate: CPU model not supported [ 3.161400] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.162235] hid: raw HID events driver (C) Jiri Kosina [ 3.170763] usbcore: registered new interface driver usbhid [ 3.173217] usbhid: USB HID core driver [ 3.175385] drop_monitor: Initializing network drop monitor service [ 3.177676] Initializing XFRM netlink socket [ 3.179424] NET: Registered protocol family 10 [ 3.183483] Segment Routing with IPv6 [ 3.185357] NET: Registered protocol family 17 [ 3.188645] mpls_gso: MPLS GSO support [ 3.195861] RAS: Correctable Errors collector initialized. [ 3.198210] AVX version of gcm_enc/dec engaged. [ 3.199874] AES CTR mode by8 optimization enabled [ 3.285579] sched_clock: Marking stable (3285550772, 0)->(4230936921, -945386149) [ 3.290268] registered taskstats version 1 [ 3.292767] Loading compiled-in X.509 certificates [ 3.295250] zswap: loaded using pool lzo/zbud [ 3.321189] Key type big_key registered [ 3.334929] Key type encrypted registered [ 3.337252] ima: No TPM chip found, activating TPM-bypass! [ 3.340126] ima: Allocated hash algorithm: sha1 [ 3.342172] ima: No architecture policies found [ 3.344365] evm: Initialising EVM extended attributes: [ 3.346355] evm: security.selinux [ 3.347567] evm: security.ima [ 3.348895] evm: security.capability [ 3.350773] evm: HMAC attrs: 0x1 [ 3.353692] rtc_cmos 00:05: setting system clock to 2026-09-08 21:35:44 UTC (1788903344) [ 3.360915] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 3.364139] debug: unmapping init [mem 0xffffffff9fd82000-0xffffffffa0058fff] [ 3.373149] Write protecting the kernel read-only data: 28672k [ 3.376611] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 3.379360] debug: unmapping init [mem 0xffffffff9ed14000-0xffffffff9edfffff] [ 3.411839] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.421942] systemd[1]: Detected virtualization kvm. [ 3.424182] systemd[1]: Detected architecture x86-64. [ 3.426356] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.450589] systemd[1]: No hostname configured. [ 3.452715] systemd[1]: Set hostname to . [ 3.455232] random: systemd: uninitialized urandom read (16 bytes read) [ 3.458471] systemd[1]: Initializing machine ID from random generator. [ 3.611803] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615661] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.618844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.621826] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.626001] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.226290] device-mapper: uevent: version 1.0.3 [ 4.228605] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.566432] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.915926] virtio_net virtio0 ens2: renamed from eth0 [ 5.032187] scsi host0: ata_piix [ 5.052641] scsi host1: ata_piix [ 5.054377] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.057164] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.550359] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.863078] random: crng init done [ 9.864649] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.142746] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.264798] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.485721] SELinux: Disabled at runtime. [ 11.551946] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.561717] systemd[1]: Detected virtualization kvm. [ 11.564242] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.012499] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.014821] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.018050] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.022254] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.024616] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.030742] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.035223] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.170956] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.512864] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.758402] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.767188] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.885837] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.901935] EDAC sbridge: Ver: 1.1.2 [ 14.525809] Key type dns_resolver registered [ 14.815663] NFS: Registering the id_resolver key type [ 14.817498] Key type id_resolver registered [ 14.818978] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg152-server login: [ 40.384044] libcfs: loading out-of-tree module taints kernel. [ 40.403381] Key type ._llcrypt registered [ 40.404867] Key type .llcrypt registered [ 40.450666] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 59.252317] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 60.774330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.790796] alg: No test for adler32 (adler32-zlib) [ 62.246337] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 63.121112] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 64.925243] Key type lgssc registered [ 66.678809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.281390] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.098883] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 134.395052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.433793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.634759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 135.673611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 135.750783] Lustre: lustre-MDT0000: new disk, initializing [ 135.805024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.822886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.752631] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 146.656580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.700455] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 146.714460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 146.716390] Lustre: Skipped 1 previous similar message [ 146.759439] Lustre: lustre-MDT0001: new disk, initializing [ 146.786449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 146.796889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 146.801832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.517626] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.434749] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 157.179726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 157.315773] Lustre: lustre-OST0000: new disk, initializing [ 157.317828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 157.321856] Lustre: 8438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 157.352250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.024954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 158.031490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.074264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 160.757138] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.290848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.357515] Lustre: lustre-OST0001: new disk, initializing [ 168.359223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 168.365737] Lustre: 9515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.389920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.089194] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.092380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 174.100221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 174.135610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 178.269295] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 183.783493] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 187.262836] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing check_logdir /tmp/testlogs/ [ 189.884245] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing yml_node [ 192.081921] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 193.172957] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 194.292179] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 195.025334] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:38:56 EDT 2026 [ 203.097525] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 203.786547] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 218.591704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 218.594797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.600176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 220.129366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 220.129631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 220.134560] Lustre: Skipped 1 previous similar message [ 220.137047] Lustre: Skipped 1 previous similar message [ 222.450912] Lustre: server umount lustre-MDT0000 complete [ 224.171604] LustreError: 9514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903565 with bad export cookie 9150850166123866402 [ 224.174159] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 224.178071] LustreError: 9514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 225.249078] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 225.249078] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 225.249088] Lustre: Skipped 1 previous similar message [ 225.249284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 225.260786] Lustre: Skipped 3 previous similar messages [ 229.281620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 229.285244] Lustre: Skipped 1 previous similar message [ 230.341570] Lustre: server umount lustre-MDT0001 complete [ 231.983793] Lustre: server umount lustre-OST0000 complete [ 233.739711] Lustre: server umount lustre-OST0001 complete [ 236.186337] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 239.790568] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 255.664986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.772491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 255.784167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 255.821738] Lustre: lustre-MDT0000: new disk, initializing [ 255.856875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.866518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 257.620645] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 262.337153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.385544] Lustre: 15051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 262.504555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 264.362646] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 269.098914] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 271.450616] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 274.561219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.687314] Lustre: lustre-OST0000: new disk, initializing [ 274.689023] Lustre: Skipped 1 previous similar message [ 274.690618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 274.692725] Lustre: Skipped 2 previous similar messages [ 274.695522] Lustre: 17019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 274.732434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.734623] Lustre: Skipped 1 previous similar message [ 276.314796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 276.318911] Lustre: Skipped 1 previous similar message [ 276.321733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 276.358358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 276.934381] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 281.154636] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 282.775709] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 282.860133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 284.442157] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 284.532966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 286.691791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.693458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 286.693587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 286.695499] Lustre: Skipped 1 previous similar message [ 291.348116] Lustre: server umount lustre-OST0000 complete [ 293.343891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.347391] LustreError: Skipped 1 previous similar message [ 293.349738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.928236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.930954] Lustre: Skipped 3 previous similar messages [ 298.788110] Lustre: server umount lustre-MDT0000 complete [ 300.119160] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903641 with bad export cookie 9150850166123867942 [ 300.122141] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.123716] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 300.267569] Lustre: server umount lustre-MDT0001 complete [ 301.587983] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:40:42 (1788903642) [ 304.215743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 304.441226] LustreError: 19279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.450348] LustreError: 19279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 304.468935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 305.963611] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 309.030965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.606443] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 313.208812] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 315.097181] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 317.626599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 319.983767] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 322.438571] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 339.426448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.426659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 339.427557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 339.430766] Lustre: Skipped 2 previous similar messages [ 341.967394] Lustre: server umount lustre-OST0000 complete [ 349.465506] Lustre: server umount lustre-MDT0000 complete [ 349.664100] LustreError: 19280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.669780] LustreError: 19280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 350.769513] LustreError: 19260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903691 with bad export cookie 9150850166123868824 [ 350.772553] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.774174] LustreError: 19260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 350.879098] Lustre: server umount lustre-MDT0001 complete [ 354.676090] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 355.811916] Key type lgssc unregistered [ 355.941602] LNet: 22833:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.944224] LNetError: 22833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.960345] LNet: Removed LNI 192.168.201.152@tcp [ 356.331105] Key type .llcrypt unregistered [ 356.332366] Key type ._llcrypt unregistered [ 365.740910] Key type ._llcrypt registered [ 365.742587] Key type .llcrypt registered [ 365.783334] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 366.190646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 366.202320] alg: No test for adler32 (adler32-zlib) [ 367.107918] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 367.210967] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 368.807173] Key type lgssc registered [ 369.264176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 372.594167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 372.601585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.775192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.352567] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 378.391566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.533429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.088551] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 382.843752] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 384.829785] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 387.352530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 387.488997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.555336] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 392.200610] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 398.759177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 403.936528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 403.938515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.939966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 403.944795] Lustre: Skipped 1 previous similar message [ 408.911643] Lustre: server umount lustre-OST0000 complete [ 414.177319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 414.177687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.180887] Lustre: Skipped 1 previous similar message [ 414.182486] Lustre: Skipped 1 previous similar message [ 416.345211] Lustre: server umount lustre-MDT0000 complete [ 417.719177] LustreError: 24956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903758 with bad export cookie 3507870885490611371 [ 417.720388] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.724319] LustreError: 24956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 417.848273] Lustre: server umount lustre-MDT0001 complete [ 421.747466] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 422.830797] Key type lgssc unregistered [ 422.967345] LNet: 28192:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.969479] LNetError: 28192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 422.977267] LNet: Removed LNI 192.168.201.152@tcp [ 423.293788] Key type .llcrypt unregistered [ 423.298336] Key type ._llcrypt unregistered [ 432.632052] Key type ._llcrypt registered [ 432.633080] Key type .llcrypt registered [ 432.672912] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 433.031847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 433.051057] alg: No test for adler32 (adler32-zlib) [ 433.906965] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 434.005882] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 435.591130] Key type lgssc registered [ 435.988262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 438.735415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 438.741166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.901497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.368705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 444.584203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.728499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 446.186243] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 448.666146] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 450.601467] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.974984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.094630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.172795] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 459.897249] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.343743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 474.594453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.594942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.600305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.604337] Lustre: Skipped 1 previous similar message [ 479.012113] Lustre: server umount lustre-OST0000 complete [ 480.735944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 480.738900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.742832] Lustre: Skipped 1 previous similar message [ 480.745511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.833142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.833395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.744195] Lustre: server umount lustre-MDT0000 complete [ 488.093332] LustreError: 29563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903829 with bad export cookie 10972750970758000200 [ 488.096243] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.096985] LustreError: 29563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 488.210626] Lustre: server umount lustre-MDT0001 complete [ 491.941109] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 492.996378] Key type lgssc unregistered [ 493.125338] LNet: 33475:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.127853] LNetError: 33475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.136280] LNet: Removed LNI 192.168.201.152@tcp [ 493.430104] Key type .llcrypt unregistered [ 493.431705] Key type ._llcrypt unregistered [ 501.940287] Key type ._llcrypt registered [ 501.941421] Key type .llcrypt registered [ 501.975447] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 502.354311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 502.380148] alg: No test for adler32 (adler32-zlib) [ 503.231545] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 503.328799] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 504.903252] Key type lgssc registered [ 505.269462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 507.799689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 507.806551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.948347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.324661] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 513.159594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.289689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 514.622545] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 517.038231] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.841607] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 520.954874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.065785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.834949] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 525.169283] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 534.373923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 538.591856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 538.594765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.599023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.616801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.617245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.617646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.484322] Lustre: server umount lustre-OST0000 complete [ 543.827337] Lustre: server umount lustre-MDT0000 complete [ 544.224082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 544.226689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.233186] LustreError: 34882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.239978] LustreError: 34882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 544.737956] LustreError: 34858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.078124] LustreError: 36706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903886 with bad export cookie 11926491496725363561 [ 545.081259] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.083129] LustreError: 36706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 545.212718] Lustre: server umount lustre-MDT0001 complete [ 548.948212] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 550.080520] Key type lgssc unregistered [ 550.205605] LNet: 38414:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.208614] LNetError: 38414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.219505] LNet: Removed LNI 192.168.201.152@tcp [ 550.546797] Key type .llcrypt unregistered [ 550.547706] Key type ._llcrypt unregistered [ 559.142900] Key type ._llcrypt registered [ 559.144404] Key type .llcrypt registered [ 559.186566] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 559.611140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.622554] alg: No test for adler32 (adler32-zlib) [ 560.490886] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 560.597381] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 562.183249] Key type lgssc registered [ 562.601726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 565.192511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 565.199949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.346119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.684097] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 570.590813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.730397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 572.066305] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 574.545222] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 576.473368] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.761218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 578.882426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 580.925838] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 583.327226] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 592.229766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 597.473902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 597.474620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.476963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 597.480708] Lustre: Skipped 1 previous similar message [ 598.856415] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 598.858229] Lustre: Skipped 1 previous similar message [ 602.592290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.712945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.715186] Lustre: Skipped 3 previous similar messages [ 610.271090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 610.325094] Lustre: server umount lustre-OST0000 complete [ 627.167900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 627.171280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.175149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 627.176871] Lustre: Skipped 1 previous similar message [ 628.192526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.395789] Lustre: server umount lustre-MDT0000 complete [ 630.523063] LustreError: 39783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903971 with bad export cookie 4222758190652927146 [ 630.525785] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.526184] LustreError: 39783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 636.716077] Lustre: server umount lustre-MDT0001 complete [ 640.289723] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 641.447560] Key type lgssc unregistered [ 641.571363] LNet: 43530:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.574659] LNetError: 43530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.587421] LNet: Removed LNI 192.168.201.152@tcp [ 641.923149] Key type .llcrypt unregistered [ 641.924043] Key type ._llcrypt unregistered [ 650.102521] Key type ._llcrypt registered [ 650.103406] Key type .llcrypt registered [ 650.137608] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 650.582059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.597145] alg: No test for adler32 (adler32-zlib) [ 651.433664] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 651.512540] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 653.079215] Key type lgssc registered [ 653.437136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 655.877713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 655.884766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.013638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.228189] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 660.766792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.874418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 662.089523] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 664.288067] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 665.938186] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 668.024663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.118943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.895767] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 672.163037] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 682.470352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 686.559897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 686.563756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.568668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.584744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.586238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.282783] Lustre: server umount lustre-MDT0000 complete [ 692.537132] LustreError: 44899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904033 with bad export cookie 6286413986530681885 [ 692.540231] LustreError: 44899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 692.540462] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.703856] LustreError: 44943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.704138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.711504] LustreError: 44943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 692.715214] Lustre: Skipped 1 previous similar message [ 692.715783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 692.721245] Lustre: Skipped 3 previous similar messages [ 697.823880] LustreError: 47409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.824203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 697.828023] LustreError: 47409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 697.829602] Lustre: Skipped 1 previous similar message [ 698.725248] Lustre: server umount lustre-MDT0001 complete [ 716.498394] Lustre: server umount lustre-OST0000 complete [ 721.600330] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 722.680867] Key type lgssc unregistered [ 722.812332] LNet: 48796:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.814407] LNetError: 48796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.822280] LNet: Removed LNI 192.168.201.152@tcp [ 723.142709] Key type .llcrypt unregistered [ 723.144468] Key type ._llcrypt unregistered [ 732.430150] Key type ._llcrypt registered [ 732.431581] Key type .llcrypt registered [ 732.466406] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 732.764365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.771061] alg: No test for adler32 (adler32-zlib) [ 733.622694] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 733.712399] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 735.295157] Key type lgssc registered [ 735.610304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 738.023929] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 738.031404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 753.439288] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 758.559293] LustreError: 50159:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 785.183184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 786.985382] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 789.165065] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 958.015240] Lustre: server umount lustre-OST0000 complete [ 974.146534] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 976.855556] Key type lgssc unregistered [ 977.158553] LNet: 51836:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.166136] LNetError: 51836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.219702] LNet: Removed LNI 192.168.201.152@tcp [ 979.421741] Key type .llcrypt unregistered [ 979.423388] Key type ._llcrypt unregistered [ 1007.181169] Key type ._llcrypt registered [ 1007.183053] Key type .llcrypt registered [ 1007.339760] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1008.655598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.717245] alg: No test for adler32 (adler32-zlib) [ 1010.037159] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1010.227102] hrtimer: interrupt took 3124161 ns [ 1010.324051] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1011.975191] Key type lgssc registered [ 1013.288740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.682849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.702174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.179044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.140244] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1033.701733] LustreError: 53252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.729528] LustreError: 53252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1037.128345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.401394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.858626] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1049.756762] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1055.904454] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1063.004557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.384115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.486818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1069.852527] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1078.613878] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1082.596257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.596566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.738691] Lustre: server umount lustre-OST0000 complete [ 1093.099155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.116563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.141602] Lustre: Skipped 1 previous similar message [ 1093.153381] Lustre: Skipped 2 previous similar messages [ 1096.897940] Lustre: server umount lustre-MDT0000 complete [ 1098.211595] LustreError: 53253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.232366] LustreError: 53253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1100.802993] LustreError: 53232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904441 with bad export cookie 14191244567754261021 [ 1100.814848] LustreError: 53232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1100.824992] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1101.275679] Lustre: server umount lustre-MDT0001 complete [ 1112.421487] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1115.327282] Key type lgssc unregistered [ 1115.597465] LNet: 56782:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.610306] LNetError: 56782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.629969] LNet: Removed LNI 192.168.201.152@tcp [ 1116.543213] Key type .llcrypt unregistered [ 1116.545602] Key type ._llcrypt unregistered [ 1139.822967] Key type ._llcrypt registered [ 1139.830372] Key type .llcrypt registered [ 1139.959543] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1141.354386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.381101] alg: No test for adler32 (adler32-zlib) [ 1142.506272] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1142.782645] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1144.479180] Key type lgssc registered [ 1145.883099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1154.860821] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1154.883676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.402131] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1175.519521] LustreError: 58169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 1202.271306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.826904] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1218.596423] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1225.583848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.760905] Lustre: MGS: Not available for connect from 0@lo (not set up) [ 1226.280085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.353603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1231.343764] LustreError: 58194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.362406] LustreError: 58194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1231.383679] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1236.456823] LustreError: 58195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.488587] LustreError: 59206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.501660] LustreError: 59206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1240.723466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.046572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1245.868577] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1253.656525] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1259.878977] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1262.054126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1262.068351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.086831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.659867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.661689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1266.678060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.805631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1271.810699] Lustre: Skipped 1 previous similar message [ 1276.383306] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1276.572595] Lustre: server umount lustre-OST0000 complete [ 1299.671277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.690904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.693318] Lustre: Skipped 1 previous similar message [ 1306.108502] Lustre: server umount lustre-MDT0000 complete [ 1308.156017] LustreError: 60902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.183105] LustreError: 60902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1310.198900] LustreError: 58175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904651 with bad export cookie 11997091033084522787 [ 1310.199501] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.210799] LustreError: 58175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1317.067241] Lustre: server umount lustre-MDT0001 complete [ 1328.373172] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1330.513692] Key type lgssc unregistered [ 1330.782045] LNet: 61916:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.788634] LNetError: 61916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.801944] LNet: Removed LNI 192.168.201.152@tcp [ 1331.472755] Key type .llcrypt unregistered [ 1331.473806] Key type ._llcrypt unregistered [ 1358.824803] Key type ._llcrypt registered [ 1358.828669] Key type .llcrypt registered [ 1358.917133] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1360.687603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1360.736158] alg: No test for adler32 (adler32-zlib) [ 1362.206488] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1362.639572] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1364.407225] Key type lgssc registered [ 1366.134909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.455170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.481132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.980394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.494316] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1386.468758] LustreError: 63330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.481346] LustreError: 63330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1390.120910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.478741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1394.179092] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1400.032831] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.699805] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1410.668874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1410.989341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.960601] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1421.231707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1421.391629] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1436.647885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.656453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.658824] Lustre: Skipped 1 previous similar message [ 1436.666543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.767433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.772098] Lustre: Skipped 1 previous similar message [ 1442.632931] Lustre: server umount lustre-OST0000 complete [ 1446.881232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.882340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.890914] Lustre: Skipped 1 previous similar message [ 1446.896214] Lustre: Skipped 2 previous similar messages [ 1452.009153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.011873] Lustre: Skipped 1 previous similar message [ 1457.120812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1457.123783] Lustre: Skipped 1 previous similar message [ 1459.167262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.252018] Lustre: server umount lustre-MDT0000 complete [ 1461.229430] LustreError: 63311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904802 with bad export cookie 4667337609099469338 [ 1461.236945] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.376940] Lustre: server umount lustre-MDT0001 complete [ 1466.272167] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1467.723838] Key type lgssc unregistered [ 1467.955801] LNet: 66887:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.959777] LNetError: 66887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.029578] LNet: Removed LNI 192.168.201.152@tcp [ 1469.459796] Key type .llcrypt unregistered [ 1469.461225] Key type ._llcrypt unregistered [ 1483.634498] Key type ._llcrypt registered [ 1483.635614] Key type .llcrypt registered [ 1483.709752] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1484.201544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.208463] alg: No test for adler32 (adler32-zlib) [ 1485.192144] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1485.347728] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1486.975167] Key type lgssc registered [ 1487.672694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.848532] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:00:33 (1788904833) [ 1496.836087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1496.845763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.111505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.329358] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1505.336168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.554481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1508.077254] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1512.591582] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1515.563804] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1519.491813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1519.672941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.807105] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1523.756480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1526.594410] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1535.291615] Lustre: Failing over lustre-MDT0000 [ 1535.434878] Lustre: server umount lustre-MDT0000 complete [ 1536.481386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.484464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.491170] LustreError: 68569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.506274] LustreError: 68569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1539.403444] LustreError: 71068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.066371] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.072141] Lustre: Skipped 1 previous similar message [ 1540.447387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.528390] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1540.616219] Lustre: *** cfs_fail_loc=135, val=0*** [ 1540.674536] LustreError: 71667:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1540.678746] LustreError: 71667:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.152@tcp: cfg command failed: rc = -2 [ 1540.687260] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1540.687260] [ 1540.692626] LustreError: MGC192.168.201.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1540.699581] LustreError: 71655:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1540.704443] LustreError: 71655:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 1540.711855] LustreError: 71655:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 1540.714818] LustreError: 71655:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1540.729780] Lustre: server umount lustre-MDT0000 complete [ 1540.733277] LustreError: 71655:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1544.516485] LustreError: 71068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.524445] LustreError: 71068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1544.789446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.868985] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.031995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.065870] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1547.221515] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1549.635042] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1550.314867] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1550.333163] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1550.363408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1555.442762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1555.443276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.449529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.466323] Lustre: Skipped 1 previous similar message [ 1558.731790] Lustre: server umount lustre-OST0000 complete [ 1565.667249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1565.668360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.676453] LustreError: Skipped 1 previous similar message [ 1565.681954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.686901] Lustre: Skipped 2 previous similar messages [ 1570.784217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.788574] Lustre: Skipped 1 previous similar message [ 1574.884491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.888886] Lustre: Skipped 1 previous similar message [ 1575.391144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1575.487419] Lustre: server umount lustre-MDT0000 complete [ 1577.325544] LustreError: 68550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904918 with bad export cookie 388180166376285910 [ 1577.326098] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1577.330409] LustreError: 68550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1577.516896] Lustre: server umount lustre-MDT0001 complete [ 1582.255740] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1583.826466] Key type lgssc unregistered [ 1583.961704] LNet: 73442:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.964795] LNetError: 73442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.973823] LNet: Removed LNI 192.168.201.152@tcp [ 1584.439241] Key type .llcrypt unregistered [ 1584.441175] Key type ._llcrypt unregistered [ 1596.507715] Key type ._llcrypt registered [ 1596.509756] Key type .llcrypt registered [ 1596.574520] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1597.125970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.138516] alg: No test for adler32 (adler32-zlib) [ 1598.035950] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1598.142497] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1599.751215] Key type lgssc registered [ 1600.283269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1604.310606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1604.323315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.516912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.332112] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1611.346224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.553739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1613.457228] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1617.415086] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1620.273062] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1623.727161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1623.909711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.012886] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1630.060952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1630.905127] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1637.960884] Lustre: Failing over lustre-MDT0000 [ 1638.157596] Lustre: server umount lustre-MDT0000 complete [ 1641.442144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.445038] LustreError: 74835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.449997] Lustre: Skipped 1 previous similar message [ 1641.461771] LustreError: 74835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1642.320359] LustreError: 74830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.691339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.764840] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.830279] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1642.891450] LustreError: 77918:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1642.895120] LustreError: 77918:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.152@tcp: cfg command failed: rc = -9 [ 1642.898108] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1642.898108] [ 1642.901683] LustreError: MGC192.168.201.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1642.908947] LustreError: 77906:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1642.912522] LustreError: 77906:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -9 [ 1642.916928] LustreError: 77906:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 1642.919267] LustreError: 77906:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1642.934964] Lustre: server umount lustre-MDT0000 complete [ 1642.936959] LustreError: 77906:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1645.653474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.724825] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.840938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.865215] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1647.428332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1647.544555] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1651.172693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1651.184547] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1651.201357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1656.290828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.291113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1656.296490] Lustre: Skipped 1 previous similar message [ 1656.300070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.303762] Lustre: Skipped 1 previous similar message [ 1658.321376] Lustre: server umount lustre-OST0000 complete [ 1661.409314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.409909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.411180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.420915] Lustre: Skipped 1 previous similar message [ 1665.824191] Lustre: server umount lustre-MDT0000 complete [ 1666.528041] LustreError: 76030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.532811] LustreError: 76030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1667.205404] LustreError: 74816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905008 with bad export cookie 11342369194136225392 [ 1667.209476] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.209610] LustreError: 74816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1667.326526] Lustre: server umount lustre-MDT0001 complete [ 1671.123782] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1672.232275] Key type lgssc unregistered [ 1672.368359] LNet: 79695:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.371139] LNetError: 79695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.378272] LNet: Removed LNI 192.168.201.152@tcp [ 1672.717233] Key type .llcrypt unregistered [ 1672.718287] Key type ._llcrypt unregistered [ 1677.223717] Key type ._llcrypt registered [ 1677.224821] Key type .llcrypt registered [ 1677.263575] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1677.374213] Key type .llcrypt unregistered [ 1677.376188] Key type ._llcrypt unregistered [ 1686.285936] Key type ._llcrypt registered [ 1686.287021] Key type .llcrypt registered [ 1686.319430] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1686.623709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1686.631145] alg: No test for adler32 (adler32-zlib) [ 1687.482641] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1687.572049] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1689.151100] Key type lgssc registered [ 1689.492118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.966288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.972207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.132474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.462429] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1697.400302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.530779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.965866] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1701.470307] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1703.308903] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1705.625360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.756781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.800658] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1710.176488] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1719.078076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1724.385835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1724.386592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.389739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1724.393781] Lustre: Skipped 1 previous similar message [ 1724.397549] Lustre: Skipped 1 previous similar message [ 1729.505344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.507082] Lustre: Skipped 1 previous similar message [ 1729.554624] Lustre: server umount lustre-OST0000 complete [ 1733.599896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.602556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.606949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.625025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.744445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.746613] Lustre: Skipped 3 previous similar messages [ 1744.864051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.866799] Lustre: Skipped 1 previous similar message [ 1745.375135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1745.439163] Lustre: server umount lustre-MDT0000 complete [ 1746.599298] LustreError: 81552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905087 with bad export cookie 11662279239738504046 [ 1746.602266] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1746.602685] LustreError: 81552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1746.724607] Lustre: server umount lustre-MDT0001 complete [ 1750.106301] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1751.103934] Key type lgssc unregistered [ 1751.228487] LNet: 85127:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.231968] LNetError: 85127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.241288] LNet: Removed LNI 192.168.201.152@tcp [ 1751.535513] Key type .llcrypt unregistered [ 1751.536675] Key type ._llcrypt unregistered [ 1760.052040] Key type ._llcrypt registered [ 1760.054157] Key type .llcrypt registered [ 1760.093027] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1760.426997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.483587] alg: No test for adler32 (adler32-zlib) [ 1761.346260] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1761.447102] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1763.039143] Key type lgssc registered [ 1763.378789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1765.829527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1765.835641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.970892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.367734] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1771.323240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.449276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1772.846340] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1775.375373] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1777.293210] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1779.470549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1779.583807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.530655] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1783.861161] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1791.910075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 1797.091084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1797.091241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.092664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.093369] LustreError: Skipped 1 previous similar message [ 1797.100186] Lustre: Skipped 1 previous similar message [ 1798.679081] Lustre: server umount lustre-OST0000 complete [ 1802.210088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.210365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1802.211225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.211229] Lustre: Skipped 1 previous similar message [ 1802.214014] Lustre: Skipped 1 previous similar message [ 1806.035789] Lustre: server umount lustre-MDT0000 complete [ 1807.263461] LustreError: 86496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905148 with bad export cookie 13689777546317329092 [ 1807.265510] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.271931] LustreError: 86496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1807.375635] Lustre: server umount lustre-MDT0001 complete [ 1810.773122] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1811.777859] Key type lgssc unregistered [ 1811.898665] LNet: 90065:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.901517] LNetError: 90065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.913326] LNet: Removed LNI 192.168.201.152@tcp [ 1812.202036] Key type .llcrypt unregistered [ 1812.202920] Key type ._llcrypt unregistered [ 1820.826377] Key type ._llcrypt registered [ 1820.827299] Key type .llcrypt registered [ 1820.863342] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1821.177418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.209065] alg: No test for adler32 (adler32-zlib) [ 1822.061823] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1822.150553] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1823.735138] Key type lgssc registered [ 1824.098063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.530797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.535568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.669542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.888129] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1831.613898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.724985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.035027] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1835.242866] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1836.904225] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1838.968670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1839.071634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.896449] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1843.201411] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.413861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 1858.528868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1858.529300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.530665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.536486] Lustre: Skipped 1 previous similar message [ 1862.923910] Lustre: server umount lustre-OST0000 complete [ 1867.743784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.745949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.749808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.751966] Lustre: Skipped 1 previous similar message [ 1868.768107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.768642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.293410] Lustre: server umount lustre-MDT0000 complete [ 1871.481046] LustreError: 91433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905212 with bad export cookie 7015937262156776791 [ 1871.483840] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.485015] LustreError: 91433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1871.602227] Lustre: server umount lustre-MDT0001 complete [ 1875.031037] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1876.007357] Key type lgssc unregistered [ 1876.127375] LNet: 95003:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.129675] LNetError: 95003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.139280] LNet: Removed LNI 192.168.201.152@tcp [ 1876.443215] Key type .llcrypt unregistered [ 1876.444357] Key type ._llcrypt unregistered [ 1884.849679] Key type ._llcrypt registered [ 1884.850634] Key type .llcrypt registered [ 1884.884435] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1885.298446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.308579] alg: No test for adler32 (adler32-zlib) [ 1886.154726] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1886.239418] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1887.823150] Key type lgssc registered [ 1888.189457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1890.692438] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1890.698468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1906.143337] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1911.263283] LustreError: 96367:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 1937.887183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.824781] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1942.261930] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1944.179842] Lustre: server umount lustre-OST0000 complete [ 1946.774703] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:08:08 (1788905288) [ 1948.937904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.080066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.444535] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1953.214479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.349760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1954.690578] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1957.075481] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1958.862830] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1961.054045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1961.193473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.186440] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1965.472316] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1974.501701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 1979.873279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.873486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.875032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.883803] Lustre: Skipped 1 previous similar message [ 1981.521745] Lustre: server umount lustre-OST0000 complete [ 1983.967737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.969956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.975076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.976941] Lustre: Skipped 1 previous similar message [ 1984.992185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.992959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.946787] Lustre: server umount lustre-MDT0000 complete [ 1990.111936] LustreError: 98002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1990.118218] LustreError: 98002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1990.154093] LustreError: 97985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905331 with bad export cookie 11499771796650947110 [ 1990.157081] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.158108] LustreError: 97985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1990.263827] Lustre: server umount lustre-MDT0001 complete [ 1993.533958] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1994.546461] Key type lgssc unregistered [ 1994.673364] LNet: 101711:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.676538] LNetError: 101711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.687247] LNet: Removed LNI 192.168.201.152@tcp [ 1994.975700] Key type .llcrypt unregistered [ 1994.977297] Key type ._llcrypt unregistered [ 2009.490159] Key type ._llcrypt registered [ 2009.491138] Key type .llcrypt registered [ 2009.526749] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2009.869202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.894373] alg: No test for adler32 (adler32-zlib) [ 2010.737962] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2010.828553] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2012.407142] Key type lgssc registered [ 2012.771302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2015.267797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2015.275066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.426102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.748493] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2020.587146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.723254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2022.151660] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2024.495952] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2026.207772] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2028.358330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.467864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.307932] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2032.519707] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2042.791732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2045.920071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.922899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.926935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.969188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.970431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.971572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.976291] Lustre: Skipped 1 previous similar message [ 2051.784500] Lustre: server umount lustre-OST0000 complete [ 2053.088748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.089341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.093018] Lustre: Skipped 1 previous similar message [ 2058.207875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.210464] Lustre: Skipped 2 previous similar messages [ 2059.031140] Lustre: server umount lustre-MDT0000 complete [ 2060.225138] LustreError: 103852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905401 with bad export cookie 4281950927264123031 [ 2060.228200] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2060.228680] LustreError: 103852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2060.341348] Lustre: server umount lustre-MDT0001 complete [ 2063.677352] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2064.625661] Key type lgssc unregistered [ 2064.748408] LNet: 107424:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.750903] LNetError: 107424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.759301] LNet: Removed LNI 192.168.201.152@tcp [ 2065.043491] Key type .llcrypt unregistered [ 2065.045205] Key type ._llcrypt unregistered [ 2071.996771] Key type ._llcrypt registered [ 2071.998114] Key type .llcrypt registered [ 2072.034802] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2072.462418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2072.473195] alg: No test for adler32 (adler32-zlib) [ 2073.320389] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2073.405747] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2074.983124] Key type lgssc registered [ 2075.331085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2077.682814] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2077.688881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.087359] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2098.207252] LustreError: 108644:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2124.831218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.680848] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2129.063872] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2131.246024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.350283] LustreError: MGC192.168.201.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2131.355127] LustreError: 109666:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2131.357841] LustreError: 109666:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 2131.361243] LustreError: 109666:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 2131.394205] Lustre: server umount lustre-MDT0000 complete [ 2131.395641] LustreError: 109666:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2144.255770] Lustre: server umount lustre-OST0000 complete [ 2146.920841] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 2149.535768] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2160.278213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.354965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2160.364325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2160.397324] Lustre: lustre-MDT0000: new disk, initializing [ 2160.419986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.425987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2161.661467] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2165.626469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.667113] Lustre: 112453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2165.689879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2165.691932] Lustre: Skipped 1 previous similar message [ 2165.724710] Lustre: lustre-MDT0001: new disk, initializing [ 2165.753869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.764343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2165.767446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2166.990585] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2170.555308] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.356132] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2174.580642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2174.679388] Lustre: lustre-OST0000: new disk, initializing [ 2174.681069] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2174.684500] Lustre: 114417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2174.715374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.560955] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2177.972209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2177.974718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2178.005367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2180.041965] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.387700] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2181.454923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2182.759705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2182.833302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2185.485537] Lustre: server umount lustre-OST0000 complete [ 2188.256538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.257029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.260219] Lustre: Skipped 1 previous similar message [ 2193.376138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.377710] Lustre: Skipped 2 previous similar messages [ 2196.470477] Lustre: server umount lustre-MDT0000 complete [ 2197.646389] LustreError: 115961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905538 with bad export cookie 6241849094797556313 [ 2197.648931] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.650103] LustreError: 115961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2207.244485] Lustre: server umount lustre-MDT0001 complete [ 2209.788274] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:12:31 (1788905551) [ 2210.781758] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2215.610406] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 2218.318232] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2231.005727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.110438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2231.125216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2231.172819] Lustre: lustre-MDT0000: new disk, initializing [ 2231.214638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.223657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2233.194095] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2239.132856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.191130] Lustre: 119315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2239.216710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2239.218470] Lustre: Skipped 1 previous similar message [ 2239.271476] Lustre: lustre-MDT0001: new disk, initializing [ 2239.271818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2239.355672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.380373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2239.385418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2241.819951] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2247.701305] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.533837] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2254.199145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.329559] Lustre: lustre-OST0000: new disk, initializing [ 2254.332282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2254.336338] Lustre: 121282:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2257.487750] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2261.003118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2261.008984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2261.043342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2262.163757] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.002189] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2264.084553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2265.774505] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2265.871406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2271.202161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2271.206343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.207244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.216908] Lustre: Skipped 1 previous similar message [ 2271.222304] Lustre: Skipped 1 previous similar message [ 2272.761986] Lustre: server umount lustre-OST0000 complete [ 2275.296273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.299876] LustreError: Skipped 1 previous similar message [ 2275.303720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.312837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2280.781900] Lustre: server umount lustre-MDT0000 complete [ 2282.956418] LustreError: 119307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905624 with bad export cookie 6241849094797557181 [ 2282.960878] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.966421] LustreError: 119307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2283.170700] Lustre: server umount lustre-MDT0001 complete [ 2288.679403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.035188] LustreError: 123495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.044961] LustreError: 123495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2289.093573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.099855] Lustre: Skipped 1 previous similar message [ 2291.901380] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2294.242297] LustreError: 123496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.842785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.674687] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2303.256510] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2305.927538] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.368708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.110715] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2315.299966] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2328.168351] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2330.092565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2330.095803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.100073] Lustre: Skipped 1 previous similar message [ 2330.101985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.105293] Lustre: Skipped 2 previous similar messages [ 2333.155414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.516606] Lustre: server umount lustre-OST0000 complete [ 2338.272584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2338.272669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.282629] Lustre: Skipped 2 previous similar messages [ 2343.205517] Lustre: server umount lustre-MDT0000 complete [ 2343.392079] LustreError: 123495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.399211] LustreError: 123495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2344.804665] LustreError: 123476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905685 with bad export cookie 6241849094797558063 [ 2344.807737] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.810102] LustreError: 123476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2344.948174] Lustre: server umount lustre-MDT0001 complete [ 2349.450299] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2350.700691] Key type lgssc unregistered [ 2350.869097] LNet: 127141:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.873401] LNetError: 127141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.886443] LNet: Removed LNI 192.168.201.152@tcp [ 2351.246580] Key type .llcrypt unregistered [ 2351.249230] Key type ._llcrypt unregistered [ 2358.845268] Key type ._llcrypt registered [ 2358.846251] Key type .llcrypt registered [ 2358.890322] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 2365.701026] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2366.044574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.077032] alg: No test for adler32 (adler32-zlib) [ 2366.963892] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2367.064929] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2368.655157] Key type lgssc registered [ 2369.117292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.303429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2383.310780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.422166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2384.443586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2384.492725] Lustre: lustre-MDT0000: new disk, initializing [ 2384.537332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.548598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2386.311663] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2391.208043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.257187] Lustre: 130269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2391.288835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2391.291144] Lustre: Skipped 1 previous similar message [ 2391.329272] Lustre: lustre-MDT0001: new disk, initializing [ 2391.374652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.397484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2391.402630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2393.063550] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2397.424315] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2399.873705] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.811368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.920356] Lustre: lustre-OST0000: new disk, initializing [ 2402.922569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2402.926402] Lustre: 132235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2402.967310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.336213] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2409.569298] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2410.499775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2410.503452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2410.543456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2411.244053] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2411.326905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2413.005535] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2413.097874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2415.585211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.586702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.589415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.593444] Lustre: Skipped 1 previous similar message [ 2419.994118] Lustre: server umount lustre-OST0000 complete [ 2422.239978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.242539] LustreError: Skipped 1 previous similar message [ 2422.243993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.248599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.251348] Lustre: Skipped 1 previous similar message [ 2425.826976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.829407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.626427] Lustre: server umount lustre-MDT0000 complete [ 2429.022368] LustreError: 130260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905770 with bad export cookie 374336283690385806 [ 2429.025963] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.026186] LustreError: 130260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2429.153508] Lustre: server umount lustre-MDT0001 complete [ 2432.333231] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:16:13 (1788905773) [ 2434.735491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.920901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.445692] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2439.908742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.059138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.538909] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2444.370949] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2446.635561] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2450.400762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.400942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.401701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.401705] Lustre: Skipped 1 previous similar message [ 2450.411124] Lustre: Skipped 1 previous similar message [ 2453.601538] Lustre: server umount lustre-MDT0000 complete [ 2454.991539] LustreError: 135377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905796 with bad export cookie 374336283690386674 [ 2454.994151] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.996056] LustreError: 135377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2455.131503] Lustre: server umount lustre-MDT0001 complete [ 2458.525405] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:16:39 (1788905799) [ 2461.176072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2476.639545] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2481.760579] LustreError: 137052:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2507.361701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.979801] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2512.995919] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2514.021673] Lustre: server umount lustre-OST0000 complete [ 2517.676656] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:17:38 (1788905858) [ 2520.724850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.957458] LustreError: 138501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.971223] LustreError: 138501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2522.729276] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2526.175972] LustreError: 138500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.290925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.506531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2526.510499] Lustre: Skipped 1 previous similar message [ 2528.236397] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2533.938919] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2538.194378] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2544.557281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.003934] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2557.534617] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2566.672159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.679955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.682539] Lustre: Skipped 4 previous similar messages [ 2572.137492] Lustre: server umount lustre-MDT0000 complete [ 2576.867681] LustreError: 139698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.914345] LustreError: 139698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2579.446588] LustreError: 138480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905920 with bad export cookie 374336283690387325 [ 2579.459694] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.468177] LustreError: 138480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2581.117467] Lustre: server umount lustre-MDT0001 complete [ 2602.607978] Lustre: server umount lustre-OST0000 complete [ 2618.480243] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:19:18 (1788905958) [ 2629.269776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.878742] LustreError: 142157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.033930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.046400] Lustre: Skipped 1 previous similar message [ 2635.244664] LustreError: 142158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.267454] LustreError: 142158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2637.175745] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2645.473586] LustreError: 142157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.497839] LustreError: 142157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2647.519546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.864341] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2660.410449] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2666.446678] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2672.646901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2674.922499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2678.616970] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2685.432855] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2690.470174] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2690.718398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2695.321849] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2695.539402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2697.696195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2697.700606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.709086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2697.736445] Lustre: Skipped 3 previous similar messages [ 2697.747304] Lustre: Skipped 5 previous similar messages [ 2699.559687] Lustre: server umount lustre-OST0000 complete [ 2704.353045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.721386] Lustre: server umount lustre-MDT0000 complete [ 2713.056733] LustreError: 142153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.077243] LustreError: 142153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2713.588203] LustreError: 142138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906054 with bad export cookie 374336283690388382 [ 2713.599226] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.600789] LustreError: 142138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2713.976784] Lustre: server umount lustre-MDT0001 complete [ 2722.896787] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:21:02 (1788906062) [ 2730.616259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.271666] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2751.391563] LustreError: 146316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2777.119311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.129025] Lustre: Skipped 2 previous similar messages [ 2783.638199] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2790.137907] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2796.718358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.112424] LustreError: 146341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.340034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2801.619235] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2809.306754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.129526] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2821.354734] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2827.040661] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2832.423266] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2832.695358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2837.918494] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2838.180533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2840.545643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2840.559790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.578475] Lustre: Skipped 2 previous similar messages [ 2840.581335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2840.593420] Lustre: Skipped 3 previous similar messages [ 2846.532569] Lustre: server umount lustre-MDT0000 complete [ 2850.432592] LustreError: 146324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906191 with bad export cookie 374336283690389404 [ 2850.442627] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.447607] LustreError: 146324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2850.798305] Lustre: server umount lustre-MDT0001 complete [ 2864.625273] Lustre: server umount lustre-OST0000 complete [ 2872.542495] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:23:32 (1788906212) [ 2880.306956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.967613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2901.088835] LustreError: 150509:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2926.815379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.817924] Lustre: Skipped 2 previous similar messages [ 2934.105306] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2942.538736] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2949.902064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.290565] LustreError: 150534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.315841] LustreError: 150534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2950.494843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2954.341996] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2964.189610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.737893] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2977.096306] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2982.808619] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2990.643586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.879958] Lustre: lustre-OST0001: new disk, initializing [ 2990.887613] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2990.896566] Lustre: 153379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2997.885715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2997.900085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2998.019356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2998.184882] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3007.395377] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3012.220934] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3012.409923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3017.268329] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3017.520357] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3021.298839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.322338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.335556] Lustre: Skipped 2 previous similar messages [ 3021.345389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.362461] Lustre: Skipped 3 previous similar messages [ 3025.606327] Lustre: server umount lustre-OST0000 complete [ 3044.323083] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3044.591643] Lustre: server umount lustre-OST0001 complete [ 3054.669477] Lustre: server umount lustre-MDT0000 complete [ 3058.908674] LustreError: 150516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906400 with bad export cookie 374336283690390356 [ 3058.921237] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3059.310920] Lustre: server umount lustre-MDT0001 complete [ 3084.228823] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:27:04 (1788906424) [ 3086.925933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3089.582622] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3093.797377] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:27:11 (1788906431) [ 3104.759400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.998663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3105.045899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3112.812699] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3122.778214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.861376] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3122.882789] Lustre: 156947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3128.555171] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3137.238963] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3144.381782] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3152.334240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.665187] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3159.395875] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3161.241318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3168.031537] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3173.979073] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3174.278133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3179.648235] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3179.948713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3182.052123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.070080] Lustre: Skipped 5 previous similar messages [ 3182.076076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3182.080061] Lustre: Skipped 12 previous similar messages [ 3196.383543] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3196.659807] Lustre: server umount lustre-OST0000 complete [ 3218.480933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.842823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.847365] Lustre: Skipped 6 previous similar messages [ 3221.944217] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.984036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.987438] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3226.461827] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3234.273570] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3240.547196] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3240.798268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3245.746302] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3245.957272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3251.621432] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95c706de4800.ost_server_uuid 50 [ 3253.362682] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95c706de4800.ost_server_uuid in FULL state after 0 sec [ 3253.581663] LustreError: 156953:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.52@tcp) returned error from blocking AST (req@0000000086f4024f x1875803591924352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000034ea4831/0x531e8dddfe7dcae lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.52@tcp remote: 0x987069eb387f93c5 expref: 5 pid: 156952 timeout: 3353 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3253.628780] LustreError: lustre-MDT0000: A client on nid 192.168.201.52@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3253.635514] LustreError: 156942:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.52@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000034ea4831/0x531e8dddfe7dcae lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.52@tcp remote: 0x987069eb387f93c5 expref: 6 pid: 156952 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3258.865091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3258.886849] LustreError: Skipped 3 previous similar messages [ 3265.024498] Lustre: server umount lustre-OST0000 complete [ 3275.236637] Lustre: server umount lustre-MDT0000 complete [ 3278.312669] LustreError: 156958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.342241] LustreError: 156958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3279.112154] LustreError: 160095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906620 with bad export cookie 374336283690391539 [ 3279.120778] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.122143] LustreError: 160095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3285.739019] Lustre: server umount lustre-MDT0001 complete [ 3297.857262] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3301.313581] Key type lgssc unregistered [ 3301.634951] LNet: 163063:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.646684] LNetError: 163063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.663936] LNet: Removed LNI 192.168.201.152@tcp [ 3302.562141] Key type .llcrypt unregistered [ 3302.563188] Key type ._llcrypt unregistered [ 3328.337649] Key type ._llcrypt registered [ 3328.339406] Key type .llcrypt registered [ 3328.444193] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3330.217444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.275478] alg: No test for adler32 (adler32-zlib) [ 3331.463377] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3331.774322] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3333.543192] Key type lgssc registered [ 3335.090611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3343.550321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3343.563554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.060115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.767579] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3355.619092] LustreError: 164483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.657275] LustreError: 164483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3358.912506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.338975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.356817] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3373.015483] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3379.872458] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3386.585697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.070381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.130296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3394.111977] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3402.880195] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3409.838575] Lustre: Failing over lustre-MDT0000 [ 3409.895801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.904843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.911856] Lustre: Skipped 1 previous similar message [ 3409.930042] Lustre: Skipped 2 previous similar messages [ 3410.065560] Lustre: server umount lustre-MDT0000 complete [ 3410.800401] LustreError: 164477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.009257] LustreError: 164482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.039413] LustreError: 164482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3420.141181] LustreError: 165210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.168715] LustreError: 165210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3422.547521] LustreError: 164463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.52@tcp arrived at 1788906763 with bad export cookie 7536927204458638110 [ 3422.571113] LustreError: 164463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3425.248893] LustreError: 164479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.274913] LustreError: 164479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3429.941446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.074444] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.385049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.449741] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3434.694696] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3435.499549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3435.529742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3437.007611] Lustre: lustre-MDT0000: Denying connection for new client 8e7ef9ed-5e3e-4c46-97b0-de4b3486edb1 (at 192.168.201.52@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 3445.748102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.753031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.753858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.786327] Lustre: Skipped 2 previous similar messages [ 3450.851664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.858520] Lustre: Skipped 2 previous similar messages [ 3451.662697] Lustre: server umount lustre-OST0000 complete [ 3455.472813] LustreError: 168209:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3455.488747] Lustre: 167559:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3455.506972] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3455.527424] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3455.541441] LustreError: 167559:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000078539240 x1875804602543104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3455.565333] LustreError: 167559:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3455.575241] LustreError: 167559:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3455.582112] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 3455.842780] Lustre: server umount lustre-MDT0000 complete [ 3455.971112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.987203] LustreError: 164479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.470056] LustreError: 164462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906800 with bad export cookie 7536927204458638215 [ 3459.480378] LustreError: 164462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3459.480642] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.823578] Lustre: server umount lustre-MDT0001 complete [ 3472.883717] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3475.846171] Key type lgssc unregistered [ 3476.227996] LNet: 168963:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.248311] LNetError: 168963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3477.285579] LNet: Removed LNI 192.168.201.152@tcp [ 3478.134168] Key type .llcrypt unregistered [ 3478.137715] Key type ._llcrypt unregistered [ 3502.081471] Key type ._llcrypt registered [ 3502.083579] Key type .llcrypt registered [ 3502.168646] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3503.174843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3503.214779] alg: No test for adler32 (adler32-zlib) [ 3504.339640] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3504.650841] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3506.343287] Key type lgssc registered [ 3507.777052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3515.957066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3515.968058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.405865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.161485] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3527.657888] LustreError: 170376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.684958] LustreError: 170376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3530.465692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.946984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3535.558354] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3542.268845] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3547.792357] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3553.773385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3554.081647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.383978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3562.321487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3567.438652] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3575.086991] Lustre: server umount lustre-OST0000 complete [ 3582.432667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3582.439333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.448262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.948520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.963436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.066506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.077316] Lustre: Skipped 1 previous similar message [ 3588.145804] Lustre: server umount lustre-MDT0000 complete [ 3591.527055] LustreError: 170358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906932 with bad export cookie 15033933049132549689 [ 3591.543815] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3591.544667] LustreError: 170358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3601.022398] Lustre: server umount lustre-MDT0001 complete [ 3610.725349] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3612.665581] Key type lgssc unregistered [ 3612.896976] LNet: 173910:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3612.904654] LNetError: 173910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3612.920991] LNet: Removed LNI 192.168.201.152@tcp [ 3613.495745] Key type .llcrypt unregistered [ 3613.497915] Key type ._llcrypt unregistered [ 3637.235646] Key type ._llcrypt registered [ 3637.241437] Key type .llcrypt registered [ 3637.357697] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3638.286719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3638.366114] alg: No test for adler32 (adler32-zlib) [ 3639.463814] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3639.741654] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3641.519374] Key type lgssc registered [ 3642.798048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3649.867503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3649.883693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.304882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.888847] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3661.792924] LustreError: 175590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.808697] LustreError: 175590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3662.002402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.275829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3666.328210] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3673.774095] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3678.274224] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3683.955814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.227698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.339569] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3696.125642] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3696.557936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3706.850212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.852924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.857548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.865158] Lustre: Skipped 1 previous similar message [ 3709.314730] Lustre: server umount lustre-OST0000 complete [ 3713.507408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3713.514295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.524777] Lustre: Skipped 1 previous similar message [ 3713.531646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.089268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.098237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.680804] Lustre: server umount lustre-MDT0000 complete [ 3721.625818] LustreError: 175614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907062 with bad export cookie 3333206155505779482 [ 3721.635591] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.640462] LustreError: 175614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3721.855799] Lustre: server umount lustre-MDT0001 complete [ 3730.996764] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3733.194690] Key type lgssc unregistered [ 3733.409687] LNet: 179124:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3733.413912] LNetError: 179124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3733.431751] LNet: Removed LNI 192.168.201.152@tcp [ 3734.110516] Key type .llcrypt unregistered [ 3734.112507] Key type ._llcrypt unregistered [ 3754.957512] Key type ._llcrypt registered [ 3754.959675] Key type .llcrypt registered [ 3755.060232] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3755.938655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3755.959890] alg: No test for adler32 (adler32-zlib) [ 3756.988537] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3757.177232] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3758.823467] Key type lgssc registered [ 3759.854174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3768.742511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3768.758801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.080623] Lustre: *** cfs_fail_loc=135, val=0*** [ 3770.160333] LustreError: 180674:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3770.165755] LustreError: 180674:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.152@tcp: cfg command failed: rc = -2 [ 3770.182982] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3770.182982] [ 3770.192966] LustreError: MGC192.168.201.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3770.208981] LustreError: 180635:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3770.216796] LustreError: 180635:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 3770.226610] LustreError: 180635:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 3770.234072] LustreError: 180635:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3770.302871] Lustre: server umount lustre-MDT0000 complete [ 3770.307478] LustreError: 180635:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3776.228605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3776.540370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.266735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3786.733824] LustreError: 181046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.748410] LustreError: 181046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3787.193976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3787.504311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3791.617965] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3798.458989] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3803.502304] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3806.176875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.181155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.199937] Lustre: Skipped 1 previous similar message [ 3811.296705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.310506] Lustre: Skipped 2 previous similar messages [ 3811.676414] Lustre: server umount lustre-MDT0000 complete [ 3812.320889] LustreError: 182098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.076340] LustreError: 182613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907156 with bad export cookie 17497238504179358480 [ 3815.088060] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.443765] Lustre: server umount lustre-MDT0001 complete [ 3824.890019] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3827.321717] Key type lgssc unregistered [ 3827.505859] LNet: 183365:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3827.512914] LNetError: 183365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3827.525239] LNet: Removed LNI 192.168.201.152@tcp [ 3828.164865] Key type .llcrypt unregistered [ 3828.167302] Key type ._llcrypt unregistered [ 3848.251774] Key type ._llcrypt registered [ 3848.255861] Key type .llcrypt registered [ 3848.368531] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3848.672301] Key type .llcrypt unregistered [ 3848.675437] Key type ._llcrypt unregistered [ 3864.505088] Key type ._llcrypt registered [ 3864.508607] Key type .llcrypt registered [ 3864.590907] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3866.247857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3866.280327] alg: No test for adler32 (adler32-zlib) [ 3867.318779] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3867.552048] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3869.175231] Key type lgssc registered [ 3870.436447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3877.802413] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3877.817632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3893.407593] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3898.527291] LustreError: 185495:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 3925.284085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.976558] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3938.641418] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3949.918449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.485106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.523178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3955.696168] LustreError: 185520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.731788] LustreError: 185520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3960.081924] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3960.118383] LustreError: 186531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.162811] LustreError: 186531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3965.409611] LustreError: 186532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.449562] LustreError: 186532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3969.800576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.381248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3975.995394] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3984.020857] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3990.287390] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3993.254779] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4006.370949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.376517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4006.397021] Lustre: Skipped 1 previous similar message [ 4010.133714] Lustre: server umount lustre-OST0000 complete [ 4016.609332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.624886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4016.628069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4016.628075] Lustre: Skipped 1 previous similar message [ 4020.049190] Lustre: server umount lustre-MDT0000 complete [ 4021.728197] LustreError: 186536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.769603] LustreError: 186536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4024.036971] LustreError: 188159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907365 with bad export cookie 10178839801622390329 [ 4024.047551] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4024.054540] LustreError: 188159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4024.371092] Lustre: server umount lustre-MDT0001 complete [ 4036.430574] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4039.378865] Key type lgssc unregistered [ 4039.647754] LNet: 189463:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4039.657255] LNetError: 189463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4039.680180] LNet: Removed LNI 192.168.201.152@tcp [ 4040.656946] Key type .llcrypt unregistered [ 4040.658819] Key type ._llcrypt unregistered [ 4065.397591] Key type ._llcrypt registered [ 4065.398953] Key type .llcrypt registered [ 4065.472986] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4067.006045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4067.040052] alg: No test for adler32 (adler32-zlib) [ 4068.199583] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4068.498596] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4070.144060] Key type lgssc registered [ 4071.744704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4080.267316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4080.285623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.738438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4085.936984] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4090.875365] LustreError: 190879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.895636] LustreError: 190879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4094.378073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.816710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4099.380924] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4107.396858] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4112.936970] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4119.900241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.415643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4126.604170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4126.880164] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4134.315225] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4138.860908] Lustre: Failing over lustre-MDT0000 [ 4139.232634] Lustre: server umount lustre-MDT0000 complete [ 4139.940500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.953494] LustreError: 190874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.968495] Lustre: Skipped 1 previous similar message [ 4140.006227] LustreError: 190874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4141.467122] LustreError: 190874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.121629] LustreError: 190873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.143462] LustreError: 190873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4150.249791] LustreError: 191889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.270298] LustreError: 191889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4156.384501] Lustre: 190189:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788907481/real 1788907481] req@00000000bf7224d6 x1875805375324928/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788907497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4156.426553] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.068729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.505655] LustreError: 190875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.521159] LustreError: 190875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4166.634358] LustreError: 190188:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa87fb05 x1875805375332736/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4166.971509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.061909] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4167.891052] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4171.962954] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4172.274334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4172.299274] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4172.356697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4176.759294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4182.752504] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4190.183803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4190.195789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.208057] Lustre: Skipped 1 previous similar message [ 4190.213298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.151566] Lustre: server umount lustre-OST0000 complete [ 4197.857600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.864635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.864687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4197.887263] Lustre: Skipped 1 previous similar message [ 4201.560263] Lustre: server umount lustre-MDT0000 complete [ 4202.976793] LustreError: 191889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.003574] LustreError: 191889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4205.369062] LustreError: 193505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907546 with bad export cookie 4994580857151043298 [ 4205.375060] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.389896] LustreError: 193505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.941594] Lustre: server umount lustre-MDT0001 complete [ 4227.892950] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4230.160921] Key type lgssc unregistered [ 4230.362588] LNet: 195716:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4230.367789] LNetError: 195716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4230.381841] LNet: Removed LNI 192.168.201.152@tcp [ 4231.087443] Key type .llcrypt unregistered [ 4231.089630] Key type ._llcrypt unregistered [ 4254.331121] Key type ._llcrypt registered [ 4254.333604] Key type .llcrypt registered [ 4254.428255] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4255.554815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4255.635123] alg: No test for adler32 (adler32-zlib) [ 4256.752367] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4256.973351] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4258.671285] Key type lgssc registered [ 4259.770851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4267.642197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4267.670166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.111718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.795395] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4279.272550] LustreError: 197131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.289233] LustreError: 197131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4282.957346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.552986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4287.926413] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4296.119529] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4301.401718] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4308.367645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.783219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.922161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4315.903657] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4323.711549] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4326.485511] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4331.920382] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4345.015349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4349.922261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4349.935806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.953303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.029974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.039927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.430929] Lustre: server umount lustre-OST0000 complete [ 4360.170297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4360.185344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.194044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.196563] Lustre: Skipped 1 previous similar message [ 4364.258245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.259792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4364.269861] Lustre: Skipped 1 previous similar message [ 4365.594652] Lustre: server umount lustre-MDT0000 complete [ 4369.245092] LustreError: 199666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907710 with bad export cookie 7056729900024677692 [ 4369.253818] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4369.261366] LustreError: 199666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4369.727517] Lustre: server umount lustre-MDT0001 complete [ 4379.477822] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4382.035470] Key type lgssc unregistered [ 4382.301545] LNet: 200974:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4382.311042] LNetError: 200974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4382.338685] LNet: Removed LNI 192.168.201.152@tcp [ 4383.100274] Key type .llcrypt unregistered [ 4383.101669] Key type ._llcrypt unregistered [ 4398.586140] Key type ._llcrypt registered [ 4398.588413] Key type .llcrypt registered [ 4398.657452] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4399.680380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4399.732192] alg: No test for adler32 (adler32-zlib) [ 4400.744476] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4400.932129] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4402.567153] Key type lgssc registered [ 4403.650614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4409.799432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4409.815416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.470234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.618268] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4421.604601] LustreError: 202219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.624369] LustreError: 202219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4423.168959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.479819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4427.914544] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4434.724250] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4439.204578] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4444.693140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.066709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.760123] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4456.436986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4456.950710] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4462.725589] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4473.426819] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4485.099856] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4488.450735] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4502.496266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4502.502255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.505442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4502.522417] Lustre: Skipped 1 previous similar message [ 4506.490542] Lustre: server umount lustre-OST0000 complete [ 4510.691311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4510.698486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.709890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4510.712112] Lustre: Skipped 1 previous similar message [ 4512.736313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.739283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.870723] Lustre: server umount lustre-MDT0000 complete [ 4517.856436] LustreError: 202220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.872223] LustreError: 202220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4519.270727] LustreError: 204898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907860 with bad export cookie 1956483875933239707 [ 4519.278157] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.288307] LustreError: 204898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4519.607814] Lustre: server umount lustre-MDT0001 complete [ 4528.193979] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4530.587646] Key type lgssc unregistered [ 4530.892166] LNet: 207249:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4530.899016] LNetError: 207249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4530.914469] LNet: Removed LNI 192.168.201.152@tcp [ 4531.643883] Key type .llcrypt unregistered [ 4531.646304] Key type ._llcrypt unregistered [ 4550.693540] Key type ._llcrypt registered [ 4550.695627] Key type .llcrypt registered [ 4550.794813] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4551.649529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4551.753495] alg: No test for adler32 (adler32-zlib) [ 4552.832745] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4553.008656] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4554.688707] Key type lgssc registered [ 4555.772971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4563.247407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4563.264084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.571877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.954411] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4574.688641] LustreError: 208642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.714646] LustreError: 208642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4576.940649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.241652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4581.405312] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4588.331769] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4592.715774] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4598.172487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4598.608191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.494128] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4608.887627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4610.839692] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4623.331834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4623.348559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.356356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4624.360341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4624.361989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4624.372062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4624.398398] Lustre: Skipped 1 previous similar message [ 4627.905642] Lustre: server umount lustre-OST0000 complete [ 4633.568668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4633.578123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.591981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.713833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.716861] Lustre: Skipped 2 previous similar messages [ 4644.832960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4644.837925] Lustre: Skipped 2 previous similar messages [ 4645.343127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4645.518824] Lustre: server umount lustre-MDT0000 complete [ 4648.322187] LustreError: 208622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907989 with bad export cookie 6991735385951727527 [ 4648.334410] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.582082] Lustre: server umount lustre-MDT0001 complete [ 4656.185911] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4658.520178] Key type lgssc unregistered [ 4658.766970] LNet: 212324:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4658.780409] LNetError: 212324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4658.795743] LNet: Removed LNI 192.168.201.152@tcp [ 4659.417637] Key type .llcrypt unregistered [ 4659.420326] Key type ._llcrypt unregistered [ 4672.733296] Key type ._llcrypt registered [ 4672.735929] Key type .llcrypt registered [ 4672.855672] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4673.884744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4673.962020] alg: No test for adler32 (adler32-zlib) [ 4674.972629] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4675.160925] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4676.815389] Key type lgssc registered [ 4677.798592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4683.784318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4683.802676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.097896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.779435] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4695.522187] LustreError: 213570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.538595] LustreError: 213570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4696.091712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.396035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4700.387845] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4706.421974] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4710.988352] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4716.576048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.876990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.644381] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4727.589254] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4730.238300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4745.705272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4745.707505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.720362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4745.733480] Lustre: Skipped 1 previous similar message [ 4745.743675] Lustre: Skipped 1 previous similar message [ 4748.879308] Lustre: server umount lustre-OST0000 complete [ 4752.866092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4752.869297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.876618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4755.939129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4755.942566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.058070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4761.060169] Lustre: Skipped 1 previous similar message [ 4765.153761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4766.177190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4766.318305] Lustre: server umount lustre-MDT0000 complete [ 4769.251936] LustreError: 213549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908110 with bad export cookie 17363544148562950201 [ 4769.257860] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4769.260109] LustreError: 213549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4769.581570] Lustre: server umount lustre-MDT0001 complete [ 4777.964188] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4779.993455] Key type lgssc unregistered [ 4780.201624] LNet: 217254:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4780.212052] LNetError: 217254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4780.229073] LNet: Removed LNI 192.168.201.152@tcp [ 4780.860260] Key type .llcrypt unregistered [ 4780.862153] Key type ._llcrypt unregistered [ 4793.877332] Key type ._llcrypt registered [ 4793.879804] Key type .llcrypt registered [ 4793.966963] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4794.496132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4794.516779] alg: No test for adler32 (adler32-zlib) [ 4795.485807] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4795.683693] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4797.351155] Key type lgssc registered [ 4798.144832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4803.826512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4803.843743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.126397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.823212] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4814.103296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.321324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4817.585484] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4822.571684] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4826.423351] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4831.100117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.325383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.825589] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4841.482807] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.790109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4857.956794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4857.995455] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4858.140088] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4862.354395] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4867.820118] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4869.140362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4880.415842] Lustre: Permanently deactivating lustre-OST0001 [ 4880.426290] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4888.611394] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4895.979242] Lustre: Permanently reactivating lustre-OST0001 [ 4895.992239] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4895.997757] Lustre: Skipped 2 previous similar messages [ 4904.991963] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.999716] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4905.004869] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4905.023096] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4905.042355] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4908.521187] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.523989] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4908.532554] Lustre: Skipped 2 previous similar messages [ 4908.540044] Lustre: Skipped 1 previous similar message [ 4912.331278] Lustre: server umount lustre-OST0001 complete [ 4913.643621] LustreError: 220319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.674024] LustreError: 220319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4916.710279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4916.718589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.729645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4917.728402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4917.734263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4918.768302] LustreError: 220904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.785686] LustreError: 220904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4921.350741] Lustre: server umount lustre-OST0000 complete [ 4926.943949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4926.948241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.954144] Lustre: Skipped 1 previous similar message [ 4926.957080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.960158] Lustre: Skipped 2 previous similar messages [ 4930.281397] Lustre: server umount lustre-MDT0000 complete [ 4933.027780] LustreError: 222349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908274 with bad export cookie 1870686445586270464 [ 4933.036746] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.038978] LustreError: 222349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4933.248783] Lustre: server umount lustre-MDT0001 complete [ 4940.609407] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4942.874353] Key type lgssc unregistered [ 4943.150077] LNet: 223611:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4943.156778] LNetError: 223611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4943.173225] LNet: Removed LNI 192.168.201.152@tcp [ 4943.829354] Key type .llcrypt unregistered [ 4943.831361] Key type ._llcrypt unregistered [ 4973.810284] Key type ._llcrypt registered [ 4973.812795] Key type .llcrypt registered [ 4973.908524] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4974.654258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4974.743087] alg: No test for adler32 (adler32-zlib) [ 4975.737333] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4975.908517] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4977.529704] Key type lgssc registered [ 4978.371478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4984.079596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4984.101590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.277384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4985.302857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4985.438204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.237538] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4994.449749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.516563] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4994.539958] Lustre: 225794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4994.695974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4997.856768] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5003.041573] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5007.137376] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5012.445559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5012.614826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5012.785185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.091908] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5019.275517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5022.008172] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5024.238744] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5031.480415] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5041.788746] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5049.854439] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5059.288725] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5076.023350] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5076.028961] Lustre: Skipped 1 previous similar message [ 5111.270172] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5111.278571] Lustre: Skipped 4 previous similar messages [ 5178.757604] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5178.766647] Lustre: Skipped 7 previous similar messages [ 5198.303921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5198.310151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.316666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5198.821063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.822559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5198.831989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5202.680496] Lustre: server umount lustre-OST0000 complete [ 5204.960018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.966320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.976589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.056932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.059611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.176468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5214.183895] Lustre: Skipped 2 previous similar messages [ 5219.295130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5219.416858] Lustre: server umount lustre-MDT0000 complete [ 5221.233110] LustreError: 225785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908562 with bad export cookie 1486671128348134640 [ 5221.239041] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.416698] Lustre: server umount lustre-MDT0001 complete [ 5228.309560] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5231.556757] Key type lgssc unregistered [ 5232.232839] LNet: 230595:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5232.253497] LNetError: 230595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5233.319658] LNet: Removed LNI 192.168.201.152@tcp [ 5234.223799] Key type .llcrypt unregistered [ 5234.225631] Key type ._llcrypt unregistered [ 5268.495481] Key type ._llcrypt registered [ 5268.504569] Key type .llcrypt registered [ 5268.891936] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5270.237469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5270.275706] alg: No test for adler32 (adler32-zlib) [ 5271.359859] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5271.608378] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5273.287612] Key type lgssc registered [ 5274.890714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5283.024955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5283.061185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.436379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.376741] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5294.563540] LustreError: 232011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.585710] LustreError: 232011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5298.619911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5299.019119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5303.726787] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5311.239919] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5317.168954] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5323.988816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.345251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.468648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5330.352244] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5336.631922] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5354.892619] Lustre: server umount lustre-OST0000 complete [ 5360.097757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.098417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5360.111216] Lustre: Skipped 1 previous similar message [ 5365.217096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5365.224771] Lustre: Skipped 2 previous similar messages [ 5370.338305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5370.341343] Lustre: Skipped 1 previous similar message [ 5371.295268] Lustre: 231324:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788908696/real 1788908696] req@000000006d1f2352 x1875806636776704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788908712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5371.326399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.383693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5372.384676] Lustre: 231322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788908697/real 1788908697] req@00000000a5abc89a x1875806636777344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788908713 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5372.589966] Lustre: server umount lustre-MDT0000 complete [ 5375.881892] LustreError: 234591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908717 with bad export cookie 10751742426635279517 [ 5375.890354] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.895591] LustreError: 234591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5376.137509] Lustre: server umount lustre-MDT0001 complete [ 5384.270535] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5386.367524] Key type lgssc unregistered [ 5386.711595] LNet: 235706:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5386.719521] LNetError: 235706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.750559] LNet: Removed LNI 192.168.201.152@tcp [ 5387.445833] Key type .llcrypt unregistered [ 5387.452094] Key type ._llcrypt unregistered [ 5407.344203] Key type ._llcrypt registered [ 5407.349981] Key type .llcrypt registered [ 5407.444464] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5408.489455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5408.646355] alg: No test for adler32 (adler32-zlib) [ 5409.744839] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5410.021549] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5411.703230] Key type lgssc registered [ 5412.777659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5683.165371] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5685.122970] Key type lgssc unregistered [ 5685.345121] LNet: 237711:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5685.350627] LNetError: 237711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5685.366576] LNet: Removed LNI 192.168.201.152@tcp [ 5686.170578] Key type .llcrypt unregistered [ 5686.172124] Key type ._llcrypt unregistered [ 5704.762807] Key type ._llcrypt registered [ 5704.767451] Key type .llcrypt registered [ 5704.871563] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5715.543510] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5716.068121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5716.111301] alg: No test for adler32 (adler32-zlib) [ 5717.049196] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5717.182725] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5718.791172] Key type lgssc registered [ 5719.575419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5737.585977] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5542 sec ========== 19:11:18 (1788909078) [ 5738.650238] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:11:19 (1788909079) === [ 5740.895466] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:11:21 (1788909081) === [ 5765.416348] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5767.187802] Key type lgssc unregistered [ 5767.430707] LNet: 242925:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5767.436165] LNetError: 242925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5767.446756] LNet: Removed LNI 192.168.201.152@tcp [ 5768.010337] Key type .llcrypt unregistered [ 5768.013553] Key type ._llcrypt unregistered