[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 585486121 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003354] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007843] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010180] LSM: Security Framework initializing [ 0.011065] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013061] *** VALIDATE selinux *** [ 0.021711] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025701] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029138] *** VALIDATE tmpfs *** [ 0.030000] *** VALIDATE proc *** [ 0.030228] *** VALIDATE cgroup *** [ 0.031000] *** VALIDATE cgroup2 *** [ 0.031250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035033] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.040497] debug: unmapping init [mem 0xffffffffa1e59000-0xffffffffa1e60fff] [ 0.042167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.051299] rcu: Hierarchical SRCU implementation. [ 0.053647] smp: Bringing up secondary CPUs ... [ 0.054693] x86: Booting SMP configuration: [ 0.055040] .... node #0, CPUs: #1 #2 #3 [ 0.063189] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.119638] node 0 deferred pages initialised in 49ms [ 0.124106] devtmpfs: initialized [ 0.125227] x86/mm: Memory block size: 128MB [ 0.128042] gcov: version magic: 0x41383552 [ 0.130268] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.132314] pinctrl core: initialized pinctrl subsystem [ 0.133261] [ 0.134007] ************************************************************* [ 0.135021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136012] ** ** [ 0.137018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138018] ** ** [ 0.139012] ** This means that this kernel is built to expose internal ** [ 0.140018] ** IOMMU data structures, which may compromise security on ** [ 0.141014] ** your system. ** [ 0.142010] ** ** [ 0.143057] ** If you see this message and you are not debugging the ** [ 0.144013] ** kernel, report this immediately to your vendor! ** [ 0.145013] ** ** [ 0.146015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147011] ************************************************************* [ 0.148908] NET: Registered protocol family 16 [ 0.149429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.150057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.151048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.154536] cpuidle: using governor menu [ 0.156116] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160631] PCI: Using configuration type 1 for base access [ 0.163422] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177009] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187731] cryptd: max_cpu_qlen set to 1000 [ 0.192854] ACPI: Added _OSI(Module Device) [ 0.195024] ACPI: Added _OSI(Processor Device) [ 0.197010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205930] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213737] ACPI: Interpreter enabled [ 0.214050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215010] ACPI: Using IOAPIC for interrupt routing [ 0.216095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228713] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242667] acpiphp: Slot [2] registered [ 0.245103] acpiphp: Slot [5] registered [ 0.246105] acpiphp: Slot [6] registered [ 0.248137] acpiphp: Slot [7] registered [ 0.249089] acpiphp: Slot [8] registered [ 0.251142] acpiphp: Slot [9] registered [ 0.252086] acpiphp: Slot [10] registered [ 0.254144] acpiphp: Slot [3] registered [ 0.256111] acpiphp: Slot [4] registered [ 0.257087] acpiphp: Slot [11] registered [ 0.258000] acpiphp: Slot [12] registered [ 0.258000] acpiphp: Slot [13] registered [ 0.258000] acpiphp: Slot [14] registered [ 0.258000] acpiphp: Slot [15] registered [ 0.258000] acpiphp: Slot [16] registered [ 0.260073] acpiphp: Slot [17] registered [ 0.261081] acpiphp: Slot [18] registered [ 0.263084] acpiphp: Slot [19] registered [ 0.264073] acpiphp: Slot [20] registered [ 0.266073] acpiphp: Slot [21] registered [ 0.267160] acpiphp: Slot [22] registered [ 0.269065] acpiphp: Slot [23] registered [ 0.270066] acpiphp: Slot [24] registered [ 0.271073] acpiphp: Slot [25] registered [ 0.273066] acpiphp: Slot [26] registered [ 0.274292] acpiphp: Slot [27] registered [ 0.275099] acpiphp: Slot [28] registered [ 0.277087] acpiphp: Slot [29] registered [ 0.279086] acpiphp: Slot [30] registered [ 0.280064] acpiphp: Slot [31] registered [ 0.281097] PCI host bridge to bus 0000:00 [ 0.283014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.285016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.287017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.289015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.291019] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.295022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.296153] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299224] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302188] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310970] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.316051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338546] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340760] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346924] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362000] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.367930] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374514] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430018] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.451291] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477155] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506017] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.521905] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563037] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.575162] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.604035] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.616101] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.621016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.627019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.644072] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.657966] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.679018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.700020] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.715000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718504] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722369] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.724359] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.727215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.732029] iommu: Default domain type: Passthrough [ 0.734564] SCSI subsystem initialized [ 0.736139] ACPI: bus type USB registered [ 0.737106] usbcore: registered new interface driver usbfs [ 0.738072] usbcore: registered new interface driver hub [ 0.740077] usbcore: registered new device driver usb [ 0.741178] pps_core: LinuxPPS API ver. 1 registered [ 0.743011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.745075] PTP clock support registered [ 0.748116] EDAC MC: Ver: 3.0.0 [ 0.750159] PCI: Using ACPI for IRQ routing [ 0.751935] NetLabel: Initializing [ 0.754014] NetLabel: domain hash size = 128 [ 0.756009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.757096] NetLabel: unlabeled traffic allowed by default [ 0.760110] vgaarb: loaded [ 0.762517] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.766025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.773000] clocksource: Switched to clocksource kvm-clock [ 0.950776] VFS: Disk quotas dquot_6.6.0 [ 0.952353] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954943] *** VALIDATE ramfs *** [ 0.956358] *** VALIDATE hugetlbfs *** [ 0.958015] pnp: PnP ACPI init [ 0.961176] pnp: PnP ACPI: found 6 devices [ 0.976639] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980016] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.982424] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.985051] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.987797] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.990236] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.993356] NET: Registered protocol family 2 [ 0.995908] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.003976] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.009272] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.015234] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.019270] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023119] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027488] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.032658] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037485] NET: Registered protocol family 1 [ 1.043821] RPC: Registered named UNIX socket transport module. [ 1.048470] RPC: Registered udp transport module. [ 1.053895] RPC: Registered tcp transport module. [ 1.058640] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.064849] NET: Registered protocol family 44 [ 1.067070] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.070364] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.072823] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.075574] PCI: CLS 0 bytes, default 64 [ 1.077317] Unpacking initramfs... [ 3.112362] debug: unmapping init [mem 0xffff97927cc54000-0xffff97927ffbffff] [ 3.116488] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.118798] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.121797] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.672895] Initialise system trusted keyrings [ 3.674569] Key type blacklist registered [ 3.676593] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.685272] zbud: loaded [ 3.689350] *** VALIDATE nfs *** [ 3.690625] *** VALIDATE nfs4 *** [ 3.692264] pstore: using deflate compression [ 3.695930] Platform Keyring initialized [ 3.833031] NET: Registered protocol family 38 [ 3.834967] Key type asymmetric registered [ 3.836619] Asymmetric key parser 'x509' registered [ 3.838442] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.841870] io scheduler mq-deadline registered [ 3.843776] io scheduler kyber registered [ 3.845557] io scheduler bfq registered [ 3.847256] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.850092] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.853481] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.859537] ACPI: Power Button [PWRF] [ 4.059300] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.242628] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.605202] hrtimer: interrupt took 42948277 ns [ 4.865454] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.989723] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.226771] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.274299] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.322306] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.334618] Non-volatile memory driver v1.3 [ 5.336361] Linux agpgart interface v0.103 [ 5.389719] virtio_blk virtio1: [vda] 133608 512-byte logical blocks (68.4 MB/65.2 MiB) [ 5.393860] vda: detected capacity change from 0 to 68407296 [ 5.418917] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.424145] vdb: detected capacity change from 0 to 1073741824 [ 5.457863] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.460729] vdc: detected capacity change from 0 to 2621440000 [ 5.508349] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.516359] vdd: detected capacity change from 0 to 2621440000 [ 5.539155] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.542249] vde: detected capacity change from 0 to 4294967296 [ 5.568847] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.572252] vdf: detected capacity change from 0 to 4294967296 [ 5.585304] libphy: Fixed MDIO Bus: probed [ 5.598429] usbcore: registered new interface driver usbserial_generic [ 5.605926] usbserial: USB Serial support registered for generic [ 5.624738] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.640747] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.642699] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.646952] mousedev: PS/2 mouse device common for all mice [ 5.649559] rtc_cmos 00:05: RTC can wake from S4 [ 5.651064] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.660979] rtc_cmos 00:05: registered as rtc0 [ 5.666218] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.680043] intel_pstate: CPU model not supported [ 5.684261] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.690298] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.705943] hid: raw HID events driver (C) Jiri Kosina [ 5.708152] usbcore: registered new interface driver usbhid [ 5.710129] usbhid: USB HID core driver [ 5.713809] drop_monitor: Initializing network drop monitor service [ 5.716464] Initializing XFRM netlink socket [ 5.718796] NET: Registered protocol family 10 [ 5.727497] Segment Routing with IPv6 [ 5.728995] NET: Registered protocol family 17 [ 5.732570] mpls_gso: MPLS GSO support [ 5.742182] RAS: Correctable Errors collector initialized. [ 5.744281] AVX version of gcm_enc/dec engaged. [ 5.746363] AES CTR mode by8 optimization enabled [ 5.911856] sched_clock: Marking stable (5911834647, 0)->(7202618685, -1290784038) [ 5.916465] registered taskstats version 1 [ 5.920283] Loading compiled-in X.509 certificates [ 5.924589] zswap: loaded using pool lzo/zbud [ 5.968521] Key type big_key registered [ 5.989288] Key type encrypted registered [ 5.990601] ima: No TPM chip found, activating TPM-bypass! [ 5.992037] ima: Allocated hash algorithm: sha1 [ 5.997782] ima: No architecture policies found [ 5.999486] evm: Initialising EVM extended attributes: [ 6.000928] evm: security.selinux [ 6.002249] evm: security.ima [ 6.003387] evm: security.capability [ 6.004650] evm: HMAC attrs: 0x1 [ 6.009239] rtc_cmos 00:05: setting system clock to 2025-11-01 02:05:20 UTC (1761962720) [ 6.019466] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 6.023683] debug: unmapping init [mem 0xffffffffa1b82000-0xffffffffa1e58fff] [ 6.028673] Write protecting the kernel read-only data: 28672k [ 6.032442] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 6.034996] debug: unmapping init [mem 0xffffffffa0b14000-0xffffffffa0bfffff] [ 6.092431] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.115989] systemd[1]: Detected virtualization kvm. [ 6.119498] systemd[1]: Detected architecture x86-64. [ 6.123181] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.164354] systemd[1]: No hostname configured. [ 6.165722] systemd[1]: Set hostname to . [ 6.171113] random: systemd: uninitialized urandom read (16 bytes read) [ 6.173045] systemd[1]: Initializing machine ID from random generator. [ 6.579863] random: systemd: uninitialized urandom read (16 bytes read) [ 6.582814] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.592994] random: systemd: uninitialized urandom read (16 bytes read) [ 6.603224] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.609967] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.916214] device-mapper: uevent: version 1.0.3 [ 8.918730] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.434162] virtio_net virtio0 ens2: renamed from eth0 [ 12.828120] random: fast init done [ 13.790105] scsi host0: ata_piix [ 14.089185] scsi host1: ata_piix [ 14.090416] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.096538] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.522322] random: crng init done [ 17.523501] random: 7 urandom warning(s) missed due to ratelimiting [ 18.380064] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.544678] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.011804] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.810054] SELinux: Disabled at runtime. [ 22.935876] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.950165] systemd[1]: Detected virtualization kvm. [ 22.952133] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.450411] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.454760] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.469879] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.487652] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.491242] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.513369] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.540383] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ 25.043563] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 25.943249] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.875385] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.977036] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.317185] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.392412] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.939576] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.864783] NFS: Registering the id_resolver key type [ 32.867108] Key type id_resolver registered [ 32.870179] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg124-server login: [ 60.052908] libcfs: loading out-of-tree module taints kernel. [ 60.065666] Key type ._llcrypt registered [ 60.067459] Key type .llcrypt registered [ 60.096421] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 67.374417] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 67.928450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.935964] alg: No test for adler32 (adler32-zlib) [ 68.913233] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 69.183119] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 70.783368] Key type lgssc registered [ 71.333971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.826705] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 78.521309] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.044433] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.803439] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.601294] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.560059] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 98.217023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.242945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 98.252302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 99.353666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 99.367760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 99.413912] Lustre: lustre-MDT0000: new disk, initializing [ 99.448649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.458946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 100.834410] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.236356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.261167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.291922] Lustre: 6464:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 106.304719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 106.307627] Lustre: Skipped 1 previous similar message [ 106.350178] Lustre: lustre-MDT0001: new disk, initializing [ 106.374562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 106.384948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 106.389498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 107.770313] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.109497] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.541609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.570908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.671923] Lustre: lustre-OST0000: new disk, initializing [ 113.674592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.698264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.588158] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.099886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 117.102578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 117.127816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 120.716238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.737051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 120.767687] Lustre: lustre-OST0001: new disk, initializing [ 120.769357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 120.785346] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 122.601198] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.467366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 127.472124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 127.488729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 128.483064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 132.161651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 139.092374] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing check_logdir /tmp/testlogs/ [ 140.557391] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing yml_node [ 142.229921] Lustre: DEBUG MARKER: Client: 2.16.59.55 [ 143.211693] Lustre: DEBUG MARKER: MDS: 2.16.59.55 [ 144.227106] Lustre: DEBUG MARKER: OSS: 2.16.59.55 [ 144.881640] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 31 22:07:39 EDT 2025 [ 151.427554] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 151.975753] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 167.903480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 167.907030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.910723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.415671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.415939] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.423121] Lustre: Skipped 2 previous similar messages [ 173.535810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.538223] Lustre: Skipped 6 previous similar messages [ 178.143090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 178.187025] LustreError: 12124:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 178.217736] Lustre: server umount lustre-MDT0000 complete [ 179.517699] LustreError: 8357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761962894 with bad export cookie 487430515010592738 [ 179.519166] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 179.522234] LustreError: 8357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 183.775515] LustreError: 9420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 183.775691] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 183.775995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 183.781611] LustreError: 9420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 183.787266] Lustre: Skipped 1 previous similar message [ 184.800136] LustreError: 9420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 184.808209] LustreError: 9420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 185.665506] LustreError: 12327:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 185.668096] LustreError: 12327:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 185.718240] Lustre: server umount lustre-MDT0001 complete [ 187.097220] LustreError: 12529:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 187.099068] LustreError: 12529:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 187.117415] Lustre: server umount lustre-OST0000 complete [ 188.395470] Lustre: server umount lustre-OST0001 complete [ 190.531722] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 193.932905] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 197.459029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.708116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.925459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.313088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.847532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.875250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.968055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.983466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 207.024390] Lustre: lustre-MDT0000: new disk, initializing [ 207.055552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.062991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.506795] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 213.169476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.202583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 213.245538] Lustre: 14892:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.358420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 214.794108] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 218.243892] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 219.645600] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 222.200482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.224691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 222.342188] Lustre: lustre-OST0000: new disk, initializing [ 222.343832] Lustre: Skipped 1 previous similar message [ 222.346164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.349050] Lustre: Skipped 2 previous similar messages [ 222.384865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.387248] Lustre: Skipped 1 previous similar message [ 224.076476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.080663] Lustre: Skipped 1 previous similar message [ 224.083741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.147619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.622395] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 228.130754] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 229.600138] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 229.678481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 231.070514] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 231.143810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 234.466810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 234.466838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.470053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 234.473879] Lustre: Skipped 1 previous similar message [ 234.477983] Lustre: Skipped 4 previous similar messages [ 244.707100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 244.709816] Lustre: Skipped 3 previous similar messages [ 246.239140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 246.257102] LustreError: 17786:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 246.260185] LustreError: 17786:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 246.298472] Lustre: server umount lustre-OST0000 complete [ 249.311888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 249.315113] LustreError: Skipped 1 previous similar message [ 249.317193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.762249] LustreError: 17989:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 253.765040] LustreError: 17989:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 253.798051] Lustre: server umount lustre-MDT0000 complete [ 254.944428] LustreError: 15882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.951670] LustreError: 15882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 255.153414] LustreError: 15666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761962969 with bad export cookie 487430515010594257 [ 255.157142] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 255.158258] LustreError: 15666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 255.281499] Lustre: server umount lustre-MDT0001 complete [ 256.434043] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:09:31 (1761962971) [ 258.645396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.800978] LustreError: 18715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.825985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.101048] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 263.010617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 264.436343] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 266.251207] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 267.503273] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 269.857950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 271.824076] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 273.753890] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 288.737248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 288.740802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.745926] Lustre: Skipped 1 previous similar message [ 288.748208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 288.750172] Lustre: Skipped 2 previous similar messages [ 293.164253] LustreError: 20958:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 293.167568] LustreError: 20958:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 293.212188] Lustre: server umount lustre-OST0000 complete [ 300.689204] Lustre: server umount lustre-MDT0000 complete [ 302.064288] LustreError: 18695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963016 with bad export cookie 487430515010595139 [ 302.067194] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 302.071653] LustreError: 18695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 302.200962] Lustre: server umount lustre-MDT0001 complete [ 306.610185] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 307.809495] Key type lgssc unregistered [ 307.949518] LNet: 21888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.953455] LNetError: 21888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.961494] LNet: Removed LNI 192.168.201.124@tcp [ 308.318126] Key type .llcrypt unregistered [ 308.319882] Key type ._llcrypt unregistered [ 317.627931] Key type ._llcrypt registered [ 317.629600] Key type .llcrypt registered [ 317.655734] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 317.982097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 318.012866] alg: No test for adler32 (adler32-zlib) [ 318.875852] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 318.959451] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 320.543140] Key type lgssc registered [ 320.917152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 323.594358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 323.600396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.765470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.063425] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 329.073174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 329.227528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 330.528328] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 332.438825] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 333.773779] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 336.065716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 336.191443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.177748] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 340.020708] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 349.545397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 354.784885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.785230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 354.789193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 354.790826] LustreError: Skipped 1 previous similar message [ 354.792525] Lustre: Skipped 1 previous similar message [ 359.909251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 359.911142] Lustre: Skipped 1 previous similar message [ 365.028443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 365.030519] Lustre: Skipped 1 previous similar message [ 369.119117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 369.134148] LustreError: 25941:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 369.167179] Lustre: server umount lustre-OST0000 complete [ 375.264580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.264859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.265411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.273066] Lustre: Skipped 2 previous similar messages [ 376.579222] LustreError: 26144:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 376.581707] LustreError: 26144:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 376.615738] Lustre: server umount lustre-MDT0000 complete [ 377.943165] LustreError: 23231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963092 with bad export cookie 11507203803638736058 [ 377.945110] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.947304] LustreError: 23231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 378.029568] LustreError: 26345:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 378.031577] LustreError: 26345:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 378.080911] Lustre: server umount lustre-MDT0001 complete [ 381.477807] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 382.473520] Key type lgssc unregistered [ 382.599416] LNet: 26869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.602165] LNetError: 26869:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.612297] LNet: Removed LNI 192.168.201.124@tcp [ 382.903439] Key type .llcrypt unregistered [ 382.904996] Key type ._llcrypt unregistered [ 391.538304] Key type ._llcrypt registered [ 391.539556] Key type .llcrypt registered [ 391.571561] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 391.901798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 391.956735] alg: No test for adler32 (adler32-zlib) [ 392.801553] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 392.887710] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 394.471182] Key type lgssc registered [ 394.823251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 397.559144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 397.565790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.720289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.028730] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 402.912319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 403.043516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.370754] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 406.204187] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 407.413202] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 411.792446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 411.898722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.842249] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 415.958101] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 425.253887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 428.512393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.516246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.521379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.562430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.563419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.565377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.348080] LustreError: 30820:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 434.391258] Lustre: server umount lustre-OST0000 complete [ 438.751920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 438.754829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.760738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 438.765228] Lustre: Skipped 1 previous similar message [ 440.800082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.800628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.919849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.921688] Lustre: Skipped 2 previous similar messages [ 450.015360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 450.058643] LustreError: 31022:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 450.061659] LustreError: 31022:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 450.090016] Lustre: server umount lustre-MDT0000 complete [ 451.039871] LustreError: 28229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.043268] LustreError: 28229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 451.474106] LustreError: 28209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963165 with bad export cookie 8441166858027890267 [ 451.477712] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.478432] LustreError: 28209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 451.549755] LustreError: 31225:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 451.552842] LustreError: 31225:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 451.609070] Lustre: server umount lustre-MDT0001 complete [ 455.447130] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 456.562268] Key type lgssc unregistered [ 456.701511] LNet: 31750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.704340] LNetError: 31750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 456.710372] LNet: Removed LNI 192.168.201.124@tcp [ 457.075561] Key type .llcrypt unregistered [ 457.076673] Key type ._llcrypt unregistered [ 466.448176] Key type ._llcrypt registered [ 466.449547] Key type .llcrypt registered [ 466.475409] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 466.855314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.866394] alg: No test for adler32 (adler32-zlib) [ 467.718215] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 467.800958] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 469.383104] Key type lgssc registered [ 469.767081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 472.398064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 472.404428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.559167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.845467] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 477.889741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 478.017112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 479.306862] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 481.314505] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 482.707086] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 485.156107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 485.278641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.208880] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 489.079814] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 498.603974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 503.781906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.786787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.786950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.790683] Lustre: Skipped 1 previous similar message [ 504.110088] LustreError: 35360:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 504.145897] Lustre: server umount lustre-OST0000 complete [ 507.389520] LustreError: 35562:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 507.391161] LustreError: 35562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 507.414993] Lustre: server umount lustre-MDT0000 complete [ 508.588102] LustreError: 33094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963223 with bad export cookie 11382656918367883266 [ 508.591810] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.653110] LustreError: 35763:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 508.656020] LustreError: 35763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 508.704328] Lustre: server umount lustre-MDT0001 complete [ 512.189658] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 513.233798] Key type lgssc unregistered [ 513.360339] LNet: 36287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.363614] LNetError: 36287:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.370304] LNet: Removed LNI 192.168.201.124@tcp [ 513.657394] Key type .llcrypt unregistered [ 513.658650] Key type ._llcrypt unregistered [ 522.652568] Key type ._llcrypt registered [ 522.654090] Key type .llcrypt registered [ 522.684502] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 523.008612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 523.035081] alg: No test for adler32 (adler32-zlib) [ 523.894234] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 523.974303] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 525.551138] Key type lgssc registered [ 525.894755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 528.343825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 528.348942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.492663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.648462] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 533.355088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 533.468130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 534.649305] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 536.326740] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 537.492820] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 539.602852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 539.704389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.535203] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 543.329937] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 554.021569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 563.167871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 563.169972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.173941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.963353] Lustre: lustre-OST0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 564.193125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.195876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.152084] LustreError: 39895:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 567.190169] Lustre: server umount lustre-OST0000 complete [ 580.370481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.371310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 580.375617] Lustre: Skipped 1 previous similar message [ 580.377870] Lustre: Skipped 2 previous similar messages [ 585.696065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 585.697908] Lustre: Skipped 2 previous similar messages [ 586.619617] LustreError: 40270:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 586.621966] LustreError: 40270:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 586.649470] Lustre: server umount lustre-MDT0000 complete [ 587.879020] LustreError: 37629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963302 with bad export cookie 11627951636626126757 [ 587.882134] LustreError: 37629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 587.882244] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.981949] LustreError: 40471:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 593.984867] LustreError: 40471:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 594.037260] Lustre: server umount lustre-MDT0001 complete [ 597.705553] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 598.758429] Key type lgssc unregistered [ 598.881408] LNet: 40996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.884699] LNetError: 40996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.896368] LNet: Removed LNI 192.168.201.124@tcp [ 599.209160] Key type .llcrypt unregistered [ 599.210172] Key type ._llcrypt unregistered [ 608.759183] Key type ._llcrypt registered [ 608.760353] Key type .llcrypt registered [ 608.790404] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 609.121448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.129117] alg: No test for adler32 (adler32-zlib) [ 609.988917] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 610.082879] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 611.663157] Key type lgssc registered [ 612.067898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 614.905568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 614.918740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.120210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.539162] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 620.709053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 620.849960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 622.267995] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 624.308868] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 625.659171] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 628.119831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 628.266441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.304724] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 632.093287] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 640.550634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 645.602559] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.604042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.606487] Lustre: Skipped 2 previous similar messages [ 645.610041] Lustre: Skipped 2 previous similar messages [ 647.927538] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.24@tcp (stopping) [ 650.720069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.723104] Lustre: Skipped 2 previous similar messages [ 651.259862] LustreError: 44605:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 651.286774] Lustre: server umount lustre-MDT0000 complete [ 652.525103] LustreError: 42336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963367 with bad export cookie 13620341208165020952 [ 652.527941] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.529992] LustreError: 42336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 653.041963] LustreError: 43069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.047666] LustreError: 43069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 653.051136] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.24@tcp (stopping) [ 655.839886] LustreError: 42352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.840083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.887903] LustreError: 43069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.888204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 657.894531] Lustre: Skipped 1 previous similar message [ 658.621468] LustreError: 44807:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 658.624809] LustreError: 44807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 658.674268] Lustre: server umount lustre-MDT0001 complete [ 676.528877] LustreError: 45031:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 676.530368] LustreError: 45031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 676.571203] Lustre: server umount lustre-OST0000 complete [ 681.915309] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 682.929590] Key type lgssc unregistered [ 683.056381] LNet: 45861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.058609] LNetError: 45861:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.067387] LNet: Removed LNI 192.168.201.124@tcp [ 683.331906] Key type .llcrypt unregistered [ 683.332873] Key type ._llcrypt unregistered [ 692.603847] Key type ._llcrypt registered [ 692.604975] Key type .llcrypt registered [ 692.635503] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 692.936233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.984026] alg: No test for adler32 (adler32-zlib) [ 693.829924] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 693.903042] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 695.479137] Key type lgssc registered [ 695.821604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 698.268249] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 698.274712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 713.695308] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 718.815194] LustreError: 47194:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 745.439227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.248110] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 748.965787] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 911.406913] LustreError: 47896:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 911.444943] Lustre: server umount lustre-OST0000 complete [ 917.102990] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 918.215799] Key type lgssc unregistered [ 918.351393] LNet: 48724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.354206] LNetError: 48724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.363462] LNet: Removed LNI 192.168.201.124@tcp [ 918.677267] Key type .llcrypt unregistered [ 918.678470] Key type ._llcrypt unregistered [ 928.252755] Key type ._llcrypt registered [ 928.253760] Key type .llcrypt registered [ 928.284555] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 928.610384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.619261] alg: No test for adler32 (adler32-zlib) [ 929.483780] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 929.585187] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 931.175135] Key type lgssc registered [ 931.581943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 934.263139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 934.269664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.428576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.752107] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 939.640053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 939.772549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.057350] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 943.031555] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 944.333333] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 946.683468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 946.804801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.628082] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 950.381512] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 955.360994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.364184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.369958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.226387] LustreError: 52301:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 957.260426] Lustre: server umount lustre-OST0000 complete [ 960.480942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 960.481129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.481886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.492091] Lustre: Skipped 1 previous similar message [ 964.669702] LustreError: 52508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 964.671779] LustreError: 52508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 964.695279] Lustre: server umount lustre-MDT0000 complete [ 965.600192] LustreError: 50087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.605325] LustreError: 50087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 965.953339] LustreError: 50068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963680 with bad export cookie 6962608755887288674 [ 965.956379] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.956995] LustreError: 50068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 966.026824] LustreError: 52710:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 966.028802] LustreError: 52710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 966.079378] Lustre: server umount lustre-MDT0001 complete [ 969.907458] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 971.019228] Key type lgssc unregistered [ 971.152453] LNet: 53235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.155213] LNetError: 53235:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.161445] LNet: Removed LNI 192.168.201.124@tcp [ 971.477770] Key type .llcrypt unregistered [ 971.479220] Key type ._llcrypt unregistered [ 980.778655] Key type ._llcrypt registered [ 980.780016] Key type .llcrypt registered [ 980.808972] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 981.220592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.230738] alg: No test for adler32 (adler32-zlib) [ 982.088525] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 982.169703] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 983.751234] Key type lgssc registered [ 984.106502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.663097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.669554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1002.143220] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1007.263166] LustreError: 54570:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 1033.887121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.654625] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1037.341106] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1039.467719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.635535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.657704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1040.879792] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1043.762659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1043.881509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1045.178031] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1046.960053] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1048.140046] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1049.057533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.059237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.062896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.065053] Lustre: Skipped 1 previous similar message [ 1049.068499] Lustre: Skipped 1 previous similar message [ 1050.860096] LustreError: 56821:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1050.894814] Lustre: server umount lustre-OST0000 complete [ 1069.024722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.025405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.027815] Lustre: Skipped 1 previous similar message [ 1069.030814] Lustre: Skipped 1 previous similar message [ 1071.165398] LustreError: 57200:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1071.168181] LustreError: 57200:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1071.194622] Lustre: server umount lustre-MDT0000 complete [ 1072.475580] LustreError: 54578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963786 with bad export cookie 1571032687417848869 [ 1072.479909] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.480594] LustreError: 54578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1078.587297] LustreError: 57403:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1078.589139] LustreError: 57403:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1078.643238] Lustre: server umount lustre-MDT0001 complete [ 1082.424570] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1083.473681] Key type lgssc unregistered [ 1083.602311] LNet: 57929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.605296] LNetError: 57929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.615352] LNet: Removed LNI 192.168.201.124@tcp [ 1083.929446] Key type .llcrypt unregistered [ 1083.930642] Key type ._llcrypt unregistered [ 1093.137505] Key type ._llcrypt registered [ 1093.138841] Key type .llcrypt registered [ 1093.169143] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1093.589757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.601081] alg: No test for adler32 (adler32-zlib) [ 1094.474050] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1094.560648] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1096.143112] Key type lgssc registered [ 1096.537101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.473587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.482597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.662407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.137619] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1105.277806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1105.406229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1106.716147] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1108.592423] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1109.859978] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1112.241038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1112.357284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.385668] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1116.343456] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1126.694984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1130.976644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1130.980379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.985651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.001902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.002696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1132.003023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.956189] LustreError: 61533:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1134.996375] Lustre: server umount lustre-OST0000 complete [ 1137.121026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.121683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.127435] Lustre: Skipped 1 previous similar message [ 1137.129364] Lustre: Skipped 1 previous similar message [ 1142.240194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.242726] Lustre: Skipped 2 previous similar messages [ 1142.399294] LustreError: 61735:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1142.402612] LustreError: 61735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1142.430457] Lustre: server umount lustre-MDT0000 complete [ 1143.718184] LustreError: 59270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963858 with bad export cookie 18083489855520516099 [ 1143.721464] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.724087] LustreError: 59270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.793828] LustreError: 61936:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1143.796017] LustreError: 61936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1143.847971] Lustre: server umount lustre-MDT0001 complete [ 1147.661297] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1148.794844] Key type lgssc unregistered [ 1148.929438] LNet: 62460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.932254] LNetError: 62460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.945344] LNet: Removed LNI 192.168.201.124@tcp [ 1149.257242] Key type .llcrypt unregistered [ 1149.258445] Key type ._llcrypt unregistered [ 1161.850819] Key type ._llcrypt registered [ 1161.852198] Key type .llcrypt registered [ 1161.888373] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1162.291516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.346303] alg: No test for adler32 (adler32-zlib) [ 1163.204798] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1163.292168] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1164.879152] Key type lgssc registered [ 1165.277861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1168.008694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1168.016633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.182537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.582802] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1173.404817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1173.511221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.659203] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1176.355810] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1177.565434] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.806458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.910968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.691689] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1183.447446] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1195.306800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1201.540839] Lustre: Failing over lustre-MDT0000 [ 1201.693597] LustreError: 66371:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1201.726303] Lustre: server umount lustre-MDT0000 complete [ 1204.192072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1204.196089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.200327] LustreError: 64124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.206547] LustreError: 64124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1205.031961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.070190] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.125895] Lustre: *** cfs_fail_loc=135, val=0*** [ 1205.169412] LustreError: 66882:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1205.173329] LustreError: 66882:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -2 [ 1205.177795] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1205.177795] [ 1205.180591] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1205.187508] LustreError: 66870:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1205.192114] LustreError: 66870:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1205.196594] LustreError: 66870:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1205.208744] Lustre: server umount lustre-MDT0000 complete [ 1205.210055] LustreError: 66870:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1205.728320] LustreError: 64124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.729545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.736875] LustreError: 64124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1207.483458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.522400] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.616458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.639541] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1208.899702] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1209.585769] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1212.901093] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1212.908450] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1212.926787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1213.919586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1213.923442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.926992] Lustre: Skipped 1 previous similar message [ 1213.929205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.018635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.018673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1218.022940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.013128] LustreError: 67681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1220.015053] LustreError: 67681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1220.053175] Lustre: server umount lustre-OST0000 complete [ 1223.138202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1223.138220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.141514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.145788] Lustre: Skipped 1 previous similar message [ 1223.150163] Lustre: Skipped 2 previous similar messages [ 1227.456547] LustreError: 67882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1227.458679] LustreError: 67882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1227.490266] Lustre: server umount lustre-MDT0000 complete [ 1228.256336] LustreError: 66260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.261021] LustreError: 66260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1228.814483] LustreError: 64104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761963943 with bad export cookie 4158046097599603166 [ 1228.820128] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.949479] Lustre: server umount lustre-MDT0001 complete [ 1232.703621] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1233.753647] Key type lgssc unregistered [ 1233.882389] LNet: 68607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.885847] LNetError: 68607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.893371] LNet: Removed LNI 192.168.201.124@tcp [ 1234.210109] Key type .llcrypt unregistered [ 1234.211703] Key type ._llcrypt unregistered [ 1243.418426] Key type ._llcrypt registered [ 1243.419662] Key type .llcrypt registered [ 1243.448090] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1243.939413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1243.945896] alg: No test for adler32 (adler32-zlib) [ 1244.802085] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1244.888133] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1246.471169] Key type lgssc registered [ 1246.895952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1249.607494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1249.614131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.778612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.106733] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1255.019253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1255.149202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.502753] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1258.326051] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1259.592138] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1261.913244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1262.031254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.009685] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1265.926947] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1277.415328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1283.950753] Lustre: Failing over lustre-MDT0000 [ 1284.097879] LustreError: 72215:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1284.125083] Lustre: server umount lustre-MDT0000 complete [ 1286.111811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.117051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.126060] LustreError: 69968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.134086] LustreError: 69968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1286.904416] LustreError: 69962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.332712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.373163] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.430123] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1287.493267] LustreError: 72725:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1287.496767] LustreError: 72725:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -9 [ 1287.500186] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1287.500186] [ 1287.503148] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1287.507206] LustreError: 72713:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1287.510323] LustreError: 72713:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1287.513538] LustreError: 72713:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1287.523472] Lustre: server umount lustre-MDT0000 complete [ 1287.524818] LustreError: 72713:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1287.649266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.653253] Lustre: Skipped 1 previous similar message [ 1289.766642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.806856] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.904143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.925344] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1291.161404] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1292.018548] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1295.335915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1295.342941] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1295.361103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1300.450094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.452793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.455480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.456754] Lustre: Skipped 1 previous similar message [ 1302.445112] LustreError: 73528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1302.447704] LustreError: 73528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1302.487630] Lustre: server umount lustre-OST0000 complete [ 1305.568663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1305.568932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.569584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.569588] Lustre: Skipped 1 previous similar message [ 1305.578494] Lustre: Skipped 1 previous similar message [ 1310.688571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.690681] Lustre: Skipped 2 previous similar messages [ 1315.808285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.810434] Lustre: Skipped 1 previous similar message [ 1318.367130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1318.399457] LustreError: 73730:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1318.401556] LustreError: 73730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1318.427396] Lustre: server umount lustre-MDT0000 complete [ 1319.633154] LustreError: 69947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964034 with bad export cookie 9606272341600564453 [ 1319.636041] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.636984] LustreError: 69947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1319.754211] Lustre: server umount lustre-MDT0001 complete [ 1323.254377] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1324.365141] Key type lgssc unregistered [ 1324.494328] LNet: 74456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.496394] LNetError: 74456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.504339] LNet: Removed LNI 192.168.201.124@tcp [ 1324.790093] Key type .llcrypt unregistered [ 1324.791229] Key type ._llcrypt unregistered [ 1328.790695] Key type ._llcrypt registered [ 1328.792065] Key type .llcrypt registered [ 1328.820114] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1328.919268] Key type .llcrypt unregistered [ 1328.920376] Key type ._llcrypt unregistered [ 1337.680143] Key type ._llcrypt registered [ 1337.681544] Key type .llcrypt registered [ 1337.710804] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1338.064852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.120157] alg: No test for adler32 (adler32-zlib) [ 1338.986858] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1339.076538] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1340.663166] Key type lgssc registered [ 1341.045565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1343.899341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1343.906304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.065820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.348788] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1349.186231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1349.308095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.589914] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1352.403752] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1353.615757] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1355.908774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1356.025321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.865221] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1359.655227] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1371.366944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1379.807850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1379.810117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.814127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1381.858414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.858815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1381.858874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.870293] Lustre: Skipped 1 previous similar message [ 1384.428747] LustreError: 78526:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1384.468281] Lustre: server umount lustre-OST0000 complete [ 1386.976851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.977645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1386.980324] Lustre: Skipped 1 previous similar message [ 1386.983904] Lustre: Skipped 1 previous similar message [ 1392.096575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.098548] Lustre: Skipped 1 previous similar message [ 1397.216138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.217861] Lustre: Skipped 1 previous similar message [ 1400.287138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1400.319048] LustreError: 78730:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1400.320890] LustreError: 78730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1400.348125] Lustre: server umount lustre-MDT0000 complete [ 1401.630414] LustreError: 76303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964116 with bad export cookie 14447962541872552362 [ 1401.633520] LustreError: 76303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1401.634031] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.707422] LustreError: 78932:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1401.709856] LustreError: 78932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1401.762994] Lustre: server umount lustre-MDT0001 complete [ 1405.415333] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1406.419385] Key type lgssc unregistered [ 1406.547470] LNet: 79456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.550840] LNetError: 79456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.560299] LNet: Removed LNI 192.168.201.124@tcp [ 1406.864202] Key type .llcrypt unregistered [ 1406.865259] Key type ._llcrypt unregistered [ 1415.857781] Key type ._llcrypt registered [ 1415.859971] Key type .llcrypt registered [ 1415.895059] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1416.330628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.357824] alg: No test for adler32 (adler32-zlib) [ 1417.217934] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1417.313746] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1418.903107] Key type lgssc registered [ 1419.265120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1421.803423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1421.810988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.963914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.263276] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1427.079373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1427.197797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1428.521534] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1430.323245] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1431.528792] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1433.777155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1433.885814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.862305] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1437.633635] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1449.255083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1457.631906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1457.634227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.637910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.681437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1459.683032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.683403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.694250] Lustre: Skipped 1 previous similar message [ 1461.617149] LustreError: 83077:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1461.660794] Lustre: server umount lustre-OST0000 complete [ 1463.263762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1463.266286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.269888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.054859] LustreError: 83279:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1469.057077] LustreError: 83279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1469.081829] Lustre: server umount lustre-MDT0000 complete [ 1469.920522] LustreError: 81537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.925974] LustreError: 81537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1470.332447] LustreError: 80812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964184 with bad export cookie 18199431591234629320 [ 1470.335662] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1470.337756] LustreError: 80812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1470.404223] LustreError: 83480:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1470.406301] LustreError: 83480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1470.454128] Lustre: server umount lustre-MDT0001 complete [ 1474.263484] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1475.279241] Key type lgssc unregistered [ 1475.419471] LNet: 84005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.421919] LNetError: 84005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.434350] LNet: Removed LNI 192.168.201.124@tcp [ 1475.724829] Key type .llcrypt unregistered [ 1475.725967] Key type ._llcrypt unregistered [ 1484.513241] Key type ._llcrypt registered [ 1484.514458] Key type .llcrypt registered [ 1484.540407] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1484.828800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.837392] alg: No test for adler32 (adler32-zlib) [ 1485.683218] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1485.768507] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1487.343147] Key type lgssc registered [ 1487.691941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1490.208516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1490.214165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.372465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.584078] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1495.440120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.571738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1496.929724] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1498.847818] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1500.123982] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1502.438388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.559153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.408934] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1506.183276] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1516.901282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1526.240659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1526.243269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.247168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1527.266285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1527.266690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1527.266933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.278624] Lustre: Skipped 1 previous similar message [ 1531.246086] LustreError: 87612:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1531.288676] Lustre: server umount lustre-OST0000 complete [ 1536.479930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.484057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.490937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.683105] LustreError: 87817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1538.684959] LustreError: 87817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1538.712633] Lustre: server umount lustre-MDT0000 complete [ 1539.994394] LustreError: 87614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964254 with bad export cookie 2159937184527491955 [ 1539.998412] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.999325] LustreError: 87614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1540.065402] LustreError: 88021:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1540.068518] LustreError: 88021:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1540.122958] Lustre: server umount lustre-MDT0001 complete [ 1543.954896] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1545.024182] Key type lgssc unregistered [ 1545.151463] LNet: 88545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.155214] LNetError: 88545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.165373] LNet: Removed LNI 192.168.201.124@tcp [ 1545.481331] Key type .llcrypt unregistered [ 1545.482292] Key type ._llcrypt unregistered [ 1554.182488] Key type ._llcrypt registered [ 1554.183416] Key type .llcrypt registered [ 1554.209991] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1554.623348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1554.632823] alg: No test for adler32 (adler32-zlib) [ 1555.481495] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1555.556907] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1557.135171] Key type lgssc registered [ 1557.479204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1559.918361] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1559.923355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1575.327293] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1580.447168] LustreError: 89878:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 1607.071163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.912537] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1610.768791] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1612.650192] LustreError: 90727:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1612.681818] Lustre: server umount lustre-OST0000 complete [ 1615.530606] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:32:10 (1761964330) [ 1617.836295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.977502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.324152] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1622.209522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1622.334659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1623.571910] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1625.461973] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.716506] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1629.005143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1629.122764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.029684] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1632.900635] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1644.524154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1652.703950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1652.706230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.709763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1654.754514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1654.755468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1654.757326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.774125] LustreError: 93801:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1657.776122] LustreError: 93801:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1657.814104] Lustre: server umount lustre-OST0000 complete [ 1659.872486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.873115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.875856] Lustre: Skipped 1 previous similar message [ 1659.877900] Lustre: Skipped 1 previous similar message [ 1664.992084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.994216] Lustre: Skipped 2 previous similar messages [ 1665.151223] LustreError: 94004:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1665.153919] LustreError: 94004:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1665.178932] Lustre: server umount lustre-MDT0000 complete [ 1666.415298] LustreError: 91379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964380 with bad export cookie 10449250996534861045 [ 1666.418341] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.418519] LustreError: 91379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1666.540603] Lustre: server umount lustre-MDT0001 complete [ 1670.226261] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1671.249299] Key type lgssc unregistered [ 1671.383381] LNet: 94731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.387147] LNetError: 94731:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.397304] LNet: Removed LNI 192.168.201.124@tcp [ 1671.706445] Key type .llcrypt unregistered [ 1671.707499] Key type ._llcrypt unregistered [ 1681.068154] Key type ._llcrypt registered [ 1681.069668] Key type .llcrypt registered [ 1681.098653] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1681.419098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.446062] alg: No test for adler32 (adler32-zlib) [ 1682.304216] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1682.390617] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1683.975150] Key type lgssc registered [ 1684.372530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1687.137516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1687.145939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.315118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.667132] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1692.567671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1692.718475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1693.987632] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1695.848228] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1697.170248] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1699.596592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1699.701268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.559846] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1703.222469] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1714.026062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1723.360329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1723.363877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.369658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1724.386267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1724.386468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1724.386961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.393646] Lustre: Skipped 1 previous similar message [ 1727.789103] LustreError: 98340:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1727.826370] Lustre: server umount lustre-OST0000 complete [ 1729.504433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.505091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1729.509054] Lustre: Skipped 1 previous similar message [ 1734.624132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.626832] Lustre: Skipped 2 previous similar messages [ 1735.234599] LustreError: 98542:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1735.236576] LustreError: 98542:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1735.265429] Lustre: server umount lustre-MDT0000 complete [ 1736.560288] LustreError: 96072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964451 with bad export cookie 7417678798346226038 [ 1736.564125] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.565378] LustreError: 96072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1736.628060] LustreError: 98745:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1736.629981] LustreError: 98745:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1736.676896] Lustre: server umount lustre-MDT0001 complete [ 1740.562310] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1741.644187] Key type lgssc unregistered [ 1741.775331] LNet: 99268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.777762] LNetError: 99268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.784334] LNet: Removed LNI 192.168.201.124@tcp [ 1742.078089] Key type .llcrypt unregistered [ 1742.080116] Key type ._llcrypt unregistered [ 1749.605843] Key type ._llcrypt registered [ 1749.607014] Key type .llcrypt registered [ 1749.639121] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1749.976716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1750.036343] alg: No test for adler32 (adler32-zlib) [ 1750.887807] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1750.973065] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1752.551194] Key type lgssc registered [ 1752.910345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.511150] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.520665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.975276] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1776.095191] LustreError: 100459:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 1802.719155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.408831] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1806.025787] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1808.142812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.232572] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1808.236351] LustreError: 101363:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1808.239139] LustreError: 101363:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1808.242144] LustreError: 101363:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1808.270614] Lustre: server umount lustre-MDT0000 complete [ 1808.271863] LustreError: 101363:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1821.151624] LustreError: 101794:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1821.185493] Lustre: server umount lustre-OST0000 complete [ 1823.968687] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 1826.758618] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1829.613981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1831.525038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1833.343037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1835.238114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1837.507832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1837.527031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.608158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1837.619461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1837.654567] Lustre: lustre-MDT0000: new disk, initializing [ 1837.680328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.689458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1838.992961] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1843.124538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.147908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.194304] Lustre: 104109:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.228248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1843.230033] Lustre: Skipped 1 previous similar message [ 1843.269147] Lustre: lustre-MDT0001: new disk, initializing [ 1843.307801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.321309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1843.325825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1844.624129] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1847.735551] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1848.998590] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1851.309075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.333221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1851.422310] Lustre: lustre-OST0000: new disk, initializing [ 1851.424236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1851.451431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.263206] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1856.334740] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1857.752316] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1860.085789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1860.089673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1860.123436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1860.896861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1862.218587] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1862.285699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1865.186014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.186776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.187823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1865.190504] Lustre: Skipped 1 previous similar message [ 1865.192675] LustreError: Skipped 1 previous similar message [ 1868.971087] LustreError: 107022:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1868.972848] LustreError: 107022:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1869.007611] Lustre: server umount lustre-OST0000 complete [ 1870.304285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.304784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.308447] Lustre: Skipped 1 previous similar message [ 1870.310119] Lustre: Skipped 1 previous similar message [ 1875.424052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.425710] Lustre: Skipped 2 previous similar messages [ 1876.352818] LustreError: 107223:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1876.355727] LustreError: 107223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1876.381011] Lustre: server umount lustre-MDT0000 complete [ 1877.673109] LustreError: 104101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964592 with bad export cookie 8038921618420075212 [ 1877.676996] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.680105] LustreError: 104101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1877.793938] Lustre: server umount lustre-MDT0001 complete [ 1880.623283] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:36:35 (1761964595) [ 1881.719679] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1886.901664] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 1889.791681] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1892.972136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1895.053479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1897.027626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1899.044342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1901.352190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1901.374429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.461665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1901.472657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1901.510269] Lustre: lustre-MDT0000: new disk, initializing [ 1901.537992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.544368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1902.850687] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1906.987910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1907.012525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1907.042842] Lustre: 110536:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1907.062687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1907.064384] Lustre: Skipped 1 previous similar message [ 1907.098233] Lustre: lustre-MDT0001: new disk, initializing [ 1907.140163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1907.143527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1908.326290] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1911.336179] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.502788] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1914.697151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1914.723361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1914.842329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.844132] Lustre: Skipped 1 previous similar message [ 1916.820866] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1920.001418] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1921.315348] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1922.040251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1922.044696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1922.075264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1922.400904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1923.754552] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1923.836856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1927.139791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1927.140568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.143996] LustreError: Skipped 1 previous similar message [ 1927.151440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.153895] Lustre: Skipped 1 previous similar message [ 1930.605379] LustreError: 113440:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1930.607098] LustreError: 113440:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1930.642089] Lustre: server umount lustre-OST0000 complete [ 1932.256764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.257517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.259923] Lustre: Skipped 2 previous similar messages [ 1937.983838] LustreError: 113642:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1937.987268] LustreError: 113642:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1938.017596] Lustre: server umount lustre-MDT0000 complete [ 1939.349575] LustreError: 110529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964653 with bad export cookie 8038921618420076080 [ 1939.353096] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.354495] LustreError: 110529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1939.474268] Lustre: server umount lustre-MDT0001 complete [ 1942.481176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.662291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.013435] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1947.028752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1948.477787] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1950.384751] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.674700] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1954.043246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1956.106963] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 1957.865308] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1970.945737] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1972.706252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.707216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.710021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.712612] Lustre: Skipped 1 previous similar message [ 1972.714926] Lustre: Skipped 3 previous similar messages [ 1974.062084] LustreError: 116662:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1974.065358] LustreError: 116662:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1974.102211] Lustre: server umount lustre-OST0000 complete [ 1977.504137] Lustre: server umount lustre-MDT0000 complete [ 1977.824438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1977.826239] LustreError: 114315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.829602] LustreError: Skipped 1 previous similar message [ 1977.842517] LustreError: 114315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1978.805331] LustreError: 116575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964693 with bad export cookie 8038921618420076948 [ 1978.809548] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.810698] LustreError: 116575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1978.944364] Lustre: server umount lustre-MDT0001 complete [ 1982.619393] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 1983.603438] Key type lgssc unregistered [ 1983.726337] LNet: 117588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.729805] LNetError: 117588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.738263] LNet: Removed LNI 192.168.201.124@tcp [ 1984.007556] Key type .llcrypt unregistered [ 1984.009252] Key type ._llcrypt unregistered [ 1989.537447] Key type ._llcrypt registered [ 1989.538350] Key type .llcrypt registered [ 1989.562034] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 1995.642880] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 1995.984230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.001161] alg: No test for adler32 (adler32-zlib) [ 1996.840654] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 1996.922619] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 1998.495172] Key type lgssc registered [ 1998.855228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.821732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2004.044279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2006.170448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2008.375558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2010.751961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2010.770642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.775599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.879781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2011.893470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2011.937504] Lustre: lustre-MDT0000: new disk, initializing [ 2011.965890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.971883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2013.255032] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2017.388483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2017.411983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2017.442310] Lustre: 120672:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2017.466418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2017.468798] Lustre: Skipped 1 previous similar message [ 2017.504984] Lustre: lustre-MDT0001: new disk, initializing [ 2017.536847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2017.548435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2017.552883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2018.814707] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2021.969923] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2023.235046] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2025.542018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2025.565907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2025.658705] Lustre: lustre-OST0000: new disk, initializing [ 2025.660493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2025.687834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.194875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2027.197802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2027.232958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2027.542362] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2030.656337] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2032.062448] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2032.147243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2033.525704] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2033.598080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2037.730403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.730965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.731602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.733729] Lustre: Skipped 1 previous similar message [ 2037.735609] LustreError: Skipped 1 previous similar message [ 2040.302552] LustreError: 123566:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2040.343119] Lustre: server umount lustre-OST0000 complete [ 2042.848895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.849537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.853549] Lustre: Skipped 1 previous similar message [ 2042.855435] Lustre: Skipped 1 previous similar message [ 2047.808351] LustreError: 123768:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2047.811386] LustreError: 123768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2047.839869] Lustre: server umount lustre-MDT0000 complete [ 2049.096069] LustreError: 121444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964763 with bad export cookie 15043418131632811365 [ 2049.099994] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.101034] LustreError: 121444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2049.167409] LustreError: 123971:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2049.170987] LustreError: 123971:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2049.227915] Lustre: server umount lustre-MDT0001 complete [ 2052.052844] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:39:27 (1761964767) [ 2054.387282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.547369] LustreError: 124640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.554077] LustreError: 124640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2054.572051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.945072] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2058.840978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2058.972202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2060.320982] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2062.188668] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2063.403525] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2064.135626] LustreError: 124670:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2064.139857] LustreError: 124670:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2064.144837] LustreError: 124670:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2064.353233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.353389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2064.354205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2064.354210] Lustre: Skipped 1 previous similar message [ 2064.357871] Lustre: Skipped 1 previous similar message [ 2069.472341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.474544] Lustre: Skipped 2 previous similar messages [ 2070.272065] LustreError: 125968:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2070.274327] LustreError: 125968:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2070.310978] Lustre: server umount lustre-MDT0000 complete [ 2071.578254] LustreError: 124620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964786 with bad export cookie 15043418131632812233 [ 2071.581285] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.582462] LustreError: 124620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2071.685641] Lustre: server umount lustre-MDT0001 complete [ 2074.369311] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:39:49 (1761964789) [ 2076.470237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2091.935552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2097.055371] LustreError: 126814:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 2122.655234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.520479] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2126.352677] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.018289] LustreError: 127470:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2127.021810] LustreError: 127470:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2127.060586] Lustre: server umount lustre-OST0000 complete [ 2129.799115] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:40:44 (1761964844) [ 2132.010318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.160122] LustreError: 128140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.591342] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2136.453461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2137.880138] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2139.704952] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2140.963039] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.425597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2143.552937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.555186] Lustre: Skipped 2 previous similar messages [ 2145.619953] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2147.539333] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2167.264166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2167.266695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.270766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.621138] LustreError: 130386:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2170.623901] LustreError: 130386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2170.649762] Lustre: server umount lustre-MDT0000 complete [ 2171.944237] LustreError: 128119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964886 with bad export cookie 15043418131632812877 [ 2171.947748] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.947947] LustreError: 128119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2172.063163] Lustre: server umount lustre-MDT0001 complete [ 2183.575758] Lustre: server umount lustre-OST0000 complete [ 2186.413076] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:41:41 (1761964901) [ 2188.637411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.788398] LustreError: 131460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.814621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.100977] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2192.998591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2194.372591] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2196.183405] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2197.501718] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.874262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2201.974404] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2203.835554] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2205.242145] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2215.339069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2215.575312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2216.874300] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2216.938649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2218.463884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.466603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.470054] Lustre: Skipped 2 previous similar messages [ 2218.471932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.473722] Lustre: Skipped 3 previous similar messages [ 2223.597876] LustreError: 134229:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2223.600104] LustreError: 134229:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2223.633270] Lustre: server umount lustre-OST0000 complete [ 2230.998460] Lustre: server umount lustre-MDT0000 complete [ 2232.260300] LustreError: 131484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761964946 with bad export cookie 15043418131632813934 [ 2232.263609] LustreError: 131484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2232.265140] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.384052] Lustre: server umount lustre-MDT0001 complete [ 2235.154086] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:42:30 (1761964950) [ 2237.434752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2252.895508] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2258.015330] LustreError: 135277:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 2283.615247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.618956] Lustre: Skipped 2 previous similar messages [ 2285.427822] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2287.221516] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2289.509814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.656794] LustreError: 135304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.698325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2290.921409] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2293.716100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2295.058124] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2296.741338] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2297.927292] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2299.208780] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2299.277733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2300.516444] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2300.575575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2303.457610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.458393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.463090] Lustre: Skipped 4 previous similar messages [ 2303.466305] Lustre: Skipped 8 previous similar messages [ 2307.324247] LustreError: 138008:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2307.327053] LustreError: 138008:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2307.355847] Lustre: server umount lustre-MDT0000 complete [ 2308.515087] LustreError: 135286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965023 with bad export cookie 15043418131632814942 [ 2308.517916] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.517919] LustreError: 135286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2308.575845] LustreError: 136192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.621825] Lustre: server umount lustre-MDT0001 complete [ 2315.923893] Lustre: server umount lustre-OST0000 complete [ 2318.557233] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:43:53 (1761965033) [ 2320.769323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2336.223482] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2341.343419] LustreError: 139056:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 2369.020964] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2370.766544] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2373.095781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.243151] LustreError: 139083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.291792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2374.500661] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2377.399824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2378.811162] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2380.629499] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2381.833686] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2384.107517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.145921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2384.207764] Lustre: lustre-OST0001: new disk, initializing [ 2384.209609] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2386.246216] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2388.985744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2388.988717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2389.019489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2389.400748] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2390.771506] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2390.840909] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2392.147146] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2392.217640] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2393.056415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2393.058457] LustreError: Skipped 1 previous similar message [ 2393.060035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.063247] Lustre: Skipped 2 previous similar messages [ 2393.064831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.067799] Lustre: Skipped 1 previous similar message [ 2407.391156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2407.409116] LustreError: 142740:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2407.411137] LustreError: 142740:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2407.424888] Lustre: server umount lustre-OST0000 complete [ 2409.442689] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.444852] LustreError: 139074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.445071] LustreError: Skipped 1 previous similar message [ 2409.449786] LustreError: 139074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2414.802395] Lustre: server umount lustre-OST0001 complete [ 2422.230349] Lustre: server umount lustre-MDT0000 complete [ 2423.461062] LustreError: 139065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965137 with bad export cookie 15043418131632815887 [ 2423.464297] LustreError: 139065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2423.465453] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.587808] Lustre: server umount lustre-MDT0001 complete [ 2425.597719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2426.989410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2428.361318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2429.717991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2432.226855] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:45:47 (1761965147) [ 2433.046627] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2433.758315] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2434.352745] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:45:49 (1761965149) [ 2436.905124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.025528] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2437.040481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2437.146699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.150407] Lustre: Skipped 6 previous similar messages [ 2438.799599] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2442.113360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2442.151530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2442.162213] Lustre: 144984:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2443.542242] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2445.254827] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2446.462740] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2448.735506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2448.809039] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2450.624252] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2452.382240] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2452.885247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2453.754520] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2453.835238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2455.076169] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2455.150138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2458.081523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2458.082883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.083529] LustreError: Skipped 3 previous similar messages [ 2458.086895] Lustre: Skipped 5 previous similar messages [ 2458.087458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.091138] Lustre: Skipped 14 previous similar messages [ 2461.907544] Lustre: server umount lustre-OST0000 complete [ 2475.738283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2477.797901] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2477.929501] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.933054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.933149] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2477.940031] Lustre: Skipped 1 previous similar message [ 2479.653467] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2481.245264] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2481.321968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2482.685817] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2482.758424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2483.961767] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b6c04faa800.ost_server_uuid 50 [ 2484.481904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b6c04faa800.ost_server_uuid in FULL state after 0 sec [ 2484.539433] LustreError: 144990:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.24@tcp) returned error from blocking AST (req@00000000328a6261 x1847553908988672 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000000cc3f2ec/0xd0c4f51657a50854 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.24@tcp remote: 0xf1dca4179941fe15 expref: 5 pid: 147835 timeout: 2584 lvb_type: 0 [ 2484.547815] LustreError: lustre-MDT0000: A client on nid 192.168.201.24@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2484.550280] LustreError: 144979:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.24@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000000cc3f2ec/0xd0c4f51657a50854 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.24@tcp remote: 0xf1dca4179941fe15 expref: 6 pid: 147835 timeout: 0 lvb_type: 0 [ 2491.659363] Lustre: server umount lustre-OST0000 complete [ 2493.409657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2493.411778] LustreError: Skipped 2 previous similar messages [ 2507.231108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2507.293544] Lustre: server umount lustre-MDT0000 complete [ 2507.744492] LustreError: 144993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.748625] LustreError: 144993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2508.569348] LustreError: 145704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965223 with bad export cookie 15043418131632817063 [ 2508.572096] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.573667] LustreError: 145704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2514.742343] Lustre: server umount lustre-MDT0001 complete [ 2518.367747] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2519.441298] Key type lgssc unregistered [ 2519.578485] LNet: 150385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.581661] LNetError: 150385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.592496] LNet: Removed LNI 192.168.201.124@tcp [ 2519.980184] Key type .llcrypt unregistered [ 2519.981775] Key type ._llcrypt unregistered [ 2528.985944] Key type ._llcrypt registered [ 2528.988223] Key type .llcrypt registered [ 2529.020609] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2529.402785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2529.443942] alg: No test for adler32 (adler32-zlib) [ 2530.290937] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2530.372527] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2531.951141] Key type lgssc registered [ 2532.304550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2534.883690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2534.894861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.103169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.348034] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2540.184239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2540.306848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2541.610040] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2543.661023] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.861343] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2546.999339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2547.135710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.828466] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2550.530752] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2562.535571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2568.492884] Lustre: Failing over lustre-MDT0000 [ 2568.683149] LustreError: 153992:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2568.721290] Lustre: server umount lustre-MDT0000 complete [ 2571.231879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2571.234031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.237892] LustreError: 151746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.242438] LustreError: 151746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2572.770625] LustreError: 151764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.770866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.775514] LustreError: 151764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2572.779983] Lustre: Skipped 1 previous similar message [ 2577.888133] LustreError: 151764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.893532] LustreError: 151764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2579.577049] LustreError: 153351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.24@tcp arrived at 1761965294 with bad export cookie 2300556928822299258 [ 2579.584712] LustreError: 153351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.209477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.250208] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.342039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.363659] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2583.569638] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2584.278508] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2584.280922] Lustre: lustre-MDT0000: Denying connection for new client 8911219d-3ae8-43cb-8d11-22cb7f8f3514 (at 192.168.201.24@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2587.621532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2591.200136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.202219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.206305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.738521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.741021] Lustre: Skipped 1 previous similar message [ 2597.105131] LustreError: 154918:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2597.109497] LustreError: 154918:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2597.159854] Lustre: server umount lustre-OST0000 complete [ 2598.417818] LustreError: 155119:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2598.420378] Lustre: 154493:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2598.422895] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2598.428596] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2598.432959] LustreError: 154493:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000891f73fa x1847554467730432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2598.439107] LustreError: 154493:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2598.441807] LustreError: 154493:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2598.445187] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2598.525971] LustreError: 155119:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2598.528696] LustreError: 155119:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2598.553863] Lustre: server umount lustre-MDT0000 complete [ 2599.743318] LustreError: 151728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965314 with bad export cookie 2300556928822299580 [ 2599.747312] LustreError: 151057:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2527deb x1847554467731328/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2599.747716] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.885664] LustreError: 155320:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2605.888809] LustreError: 155320:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2605.936918] Lustre: server umount lustre-MDT0001 complete [ 2609.578547] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2610.524149] Key type lgssc unregistered [ 2610.648356] LNet: 155845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.650630] LNetError: 155845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.660305] LNet: Removed LNI 192.168.201.124@tcp [ 2610.936829] Key type .llcrypt unregistered [ 2610.937917] Key type ._llcrypt unregistered [ 2619.611345] Key type ._llcrypt registered [ 2619.612417] Key type .llcrypt registered [ 2619.642353] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2619.954944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.002638] alg: No test for adler32 (adler32-zlib) [ 2620.854485] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2620.936390] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2622.519183] Key type lgssc registered [ 2622.881180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2625.521473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2625.526967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.698926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.874813] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2630.678560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2630.822842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.091605] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2633.806344] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2634.954932] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2637.196792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2637.281352] LustreError: 158790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.289724] LustreError: 158790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2637.321433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.964084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2639.161926] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2640.914504] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2644.448129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2644.451399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.457038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.497081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.498776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.499568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.131769] LustreError: 159430:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2649.164684] Lustre: server umount lustre-OST0000 complete [ 2651.617209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.617504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.627231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.629041] Lustre: Skipped 2 previous similar messages [ 2656.509567] LustreError: 159632:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2656.511894] LustreError: 159632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2656.544465] Lustre: server umount lustre-MDT0000 complete [ 2656.736258] LustreError: 157205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.743555] LustreError: 157205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2657.770126] LustreError: 157915:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965372 with bad export cookie 8544992663715513833 [ 2657.773235] LustreError: 157915:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2657.773748] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.836259] LustreError: 159834:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2657.838084] LustreError: 159834:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2657.892197] Lustre: server umount lustre-MDT0001 complete [ 2661.551827] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2662.591960] Key type lgssc unregistered [ 2662.728523] LNet: 160358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.730560] LNetError: 160358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.741415] LNet: Removed LNI 192.168.201.124@tcp [ 2663.048563] Key type .llcrypt unregistered [ 2663.049592] Key type ._llcrypt unregistered [ 2674.476498] Key type ._llcrypt registered [ 2674.478033] Key type .llcrypt registered [ 2674.504494] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2674.802256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.808932] alg: No test for adler32 (adler32-zlib) [ 2675.650602] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2675.728447] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2677.303136] Key type lgssc registered [ 2677.646117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2680.062659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2680.069624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.236453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.537314] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2685.398894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2685.544479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.806939] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2688.562421] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2689.689393] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2691.862995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2691.989449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.845288] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2695.611774] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2706.343297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2716.642506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.643402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.649602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.650749] LustreError: Skipped 1 previous similar message [ 2716.652463] Lustre: Skipped 1 previous similar message [ 2718.702120] LustreError: 164255:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2718.734509] Lustre: server umount lustre-OST0000 complete [ 2721.247872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.250548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.254771] Lustre: Skipped 1 previous similar message [ 2721.256777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.012826] LustreError: 164457:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2726.014562] LustreError: 164457:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2726.041231] Lustre: server umount lustre-MDT0000 complete [ 2726.880969] LustreError: 162027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.886441] LustreError: 162027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2727.303520] LustreError: 164459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965441 with bad export cookie 12265124104675531869 [ 2727.308706] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.312423] LustreError: 164459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2727.377574] LustreError: 164661:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2727.379898] LustreError: 164661:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2727.441083] Lustre: server umount lustre-MDT0001 complete [ 2730.856340] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2731.805857] Key type lgssc unregistered [ 2731.930398] LNet: 165185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2731.932638] LNetError: 165185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2731.939443] LNet: Removed LNI 192.168.201.124@tcp [ 2732.231910] Key type .llcrypt unregistered [ 2732.233111] Key type ._llcrypt unregistered [ 2740.243070] Key type ._llcrypt registered [ 2740.244338] Key type .llcrypt registered [ 2740.269068] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2740.566995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2740.637362] alg: No test for adler32 (adler32-zlib) [ 2741.488165] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2741.568726] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2743.151174] Key type lgssc registered [ 2743.529514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2747.429389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2747.439433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.580128] Lustre: *** cfs_fail_loc=135, val=0*** [ 2748.623208] LustreError: 166656:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2748.625978] LustreError: 166656:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.124@tcp: cfg command failed: rc = -2 [ 2748.629070] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2748.629070] [ 2748.632336] LustreError: MGC192.168.201.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2748.636118] LustreError: 166616:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2748.638312] LustreError: 166616:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2748.640354] LustreError: 166616:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2748.642696] LustreError: 166616:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2748.675255] Lustre: server umount lustre-MDT0000 complete [ 2748.676556] LustreError: 166616:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2751.678297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.921072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.304466] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2756.726328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2756.852768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2758.298787] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2761.032740] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2762.436729] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.203170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.204269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.211486] Lustre: Skipped 1 previous similar message [ 2767.217759] Lustre: Skipped 1 previous similar message [ 2769.216308] LustreError: 168359:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2769.218703] LustreError: 168359:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2769.261028] Lustre: server umount lustre-MDT0000 complete [ 2770.781151] LustreError: 168361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965485 with bad export cookie 13530976513161902958 [ 2770.785557] LustreError: 168361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2770.785793] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.872631] LustreError: 168563:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2770.874700] LustreError: 168563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2770.928216] Lustre: server umount lustre-MDT0001 complete [ 2776.062622] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2777.271123] Key type lgssc unregistered [ 2777.398593] LNet: 169087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2777.400863] LNetError: 169087:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2777.412458] LNet: Removed LNI 192.168.201.124@tcp [ 2777.727506] Key type .llcrypt unregistered [ 2777.728493] Key type ._llcrypt unregistered [ 2784.559436] Key type ._llcrypt registered [ 2784.560397] Key type .llcrypt registered [ 2784.586839] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2784.686244] Key type .llcrypt unregistered [ 2784.688249] Key type ._llcrypt unregistered [ 2792.282764] Key type ._llcrypt registered [ 2792.284027] Key type .llcrypt registered [ 2792.312456] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2792.764063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.776158] alg: No test for adler32 (adler32-zlib) [ 2793.649828] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2793.735893] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2795.335272] Key type lgssc registered [ 2795.914335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2799.200695] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2799.211951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2814.687278] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2819.807170] LustreError: 171138:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.124@tcp: failed processing log, type 4: rc = -110 [ 2846.431184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.269554] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2849.983534] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2852.260700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.457386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.475620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2853.691986] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2856.505376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2856.643674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2857.904243] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2859.607651] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2860.750750] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2861.713333] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2871.266624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.270474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2871.272149] Lustre: Skipped 1 previous similar message [ 2875.116138] LustreError: 173841:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2875.148285] Lustre: server umount lustre-OST0000 complete [ 2877.407947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.410149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.413287] Lustre: Skipped 1 previous similar message [ 2877.414860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.504286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.504890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.557444] LustreError: 174044:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2882.559450] LustreError: 174044:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2882.583146] Lustre: server umount lustre-MDT0000 complete [ 2883.805482] LustreError: 171147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965598 with bad export cookie 15045604281706044998 [ 2883.809221] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2883.812019] LustreError: 171147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2883.876875] LustreError: 174246:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2883.878778] LustreError: 174246:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2883.934909] Lustre: server umount lustre-MDT0001 complete [ 2887.505649] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2888.626321] Key type lgssc unregistered [ 2888.751424] LNet: 174769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2888.753950] LNetError: 174769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2888.764330] LNet: Removed LNI 192.168.201.124@tcp [ 2889.072231] Key type .llcrypt unregistered [ 2889.073970] Key type ._llcrypt unregistered [ 2897.875841] Key type ._llcrypt registered [ 2897.876903] Key type .llcrypt registered [ 2897.907367] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2898.326514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2898.345239] alg: No test for adler32 (adler32-zlib) [ 2899.198386] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2899.281266] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2900.863183] Key type lgssc registered [ 2901.225553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2903.818577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2903.825251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.003456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.318716] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2909.221259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2909.349126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2910.549242] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2912.189645] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2913.385517] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.597884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2915.726116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.777380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2917.581601] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2919.307591] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2921.458284] Lustre: Failing over lustre-MDT0000 [ 2921.532790] LustreError: 178404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2921.567318] Lustre: server umount lustre-MDT0000 complete [ 2924.004589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.005961] LustreError: 176836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.008204] Lustre: Skipped 2 previous similar messages [ 2924.012308] LustreError: 176836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2925.817920] LustreError: 178521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.120687] LustreError: 176836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.125297] LustreError: 176836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2934.241407] LustreError: 176836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.245730] LustreError: 176836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2934.440577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.487628] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.578253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.600225] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2935.885378] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2936.049838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2937.444695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2939.880335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2939.887872] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2939.908071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2943.217447] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2954.719834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.722102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.725425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.233825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.235618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2957.486125] LustreError: 179724:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2957.487876] LustreError: 179724:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2957.527086] Lustre: server umount lustre-OST0000 complete [ 2960.353520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.354367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.355720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.355727] Lustre: Skipped 1 previous similar message [ 2960.362233] Lustre: Skipped 2 previous similar messages [ 2965.472470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.474715] Lustre: Skipped 2 previous similar messages [ 2970.592069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.594036] Lustre: Skipped 1 previous similar message [ 2973.151123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2973.185559] LustreError: 179930:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2973.187113] LustreError: 179930:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2973.215692] Lustre: server umount lustre-MDT0000 complete [ 2974.416361] LustreError: 176113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965688 with bad export cookie 17068284230376774413 [ 2974.419462] LustreError: 176113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2974.420238] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.548484] Lustre: server umount lustre-MDT0001 complete [ 2977.982661] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 2978.975920] Key type lgssc unregistered [ 2979.102381] LNet: 180656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2979.104675] LNetError: 180656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2979.110456] LNet: Removed LNI 192.168.201.124@tcp [ 2979.414708] Key type .llcrypt unregistered [ 2979.415617] Key type ._llcrypt unregistered [ 2987.944680] Key type ._llcrypt registered [ 2987.945672] Key type .llcrypt registered [ 2987.974536] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 2988.395305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2988.464238] alg: No test for adler32 (adler32-zlib) [ 2989.314063] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 2989.397551] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 2990.983143] Key type lgssc registered [ 2991.322915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2993.931585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2993.937297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.107674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2996.377607] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 2999.132237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2999.270351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.528122] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3002.215902] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3003.356400] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3005.492447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3005.618805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.610408] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3009.420921] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3019.939186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3020.587440] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3024.062139] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3033.101781] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3044.832055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3044.834252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.839147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.857547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3045.859091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.859460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.865646] Lustre: Skipped 1 previous similar message [ 3048.364963] LustreError: 184440:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3048.399554] Lustre: server umount lustre-OST0000 complete [ 3050.463962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.468730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.476699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.741002] LustreError: 184641:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3055.744654] LustreError: 184641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3055.776720] Lustre: server umount lustre-MDT0000 complete [ 3056.096047] LustreError: 182723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.100563] LustreError: 182723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3056.973171] LustreError: 184220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965771 with bad export cookie 18113929213855336410 [ 3056.976641] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3056.977918] LustreError: 184220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3057.048935] LustreError: 184842:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3057.051834] LustreError: 184842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3057.108689] Lustre: server umount lustre-MDT0001 complete [ 3060.824380] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3061.862919] Key type lgssc unregistered [ 3061.992440] LNet: 185366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3061.994885] LNetError: 185366:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3062.006413] LNet: Removed LNI 192.168.201.124@tcp [ 3062.324731] Key type .llcrypt unregistered [ 3062.325813] Key type ._llcrypt unregistered [ 3071.633113] Key type ._llcrypt registered [ 3071.634081] Key type .llcrypt registered [ 3071.659776] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3071.975553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3072.021095] alg: No test for adler32 (adler32-zlib) [ 3072.878378] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3072.968652] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3074.551138] Key type lgssc registered [ 3074.906212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3077.639230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3077.644660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.818091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.191522] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3083.252818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3083.409308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3085.172478] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3087.187119] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.825847] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3091.970130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3092.142653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.626099] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3097.131070] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3104.433307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3109.560183] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3119.747230] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3125.874696] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3131.976544] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3139.552648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3139.557232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.565263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.581446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.581552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.582592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.331197] LustreError: 190651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3145.401284] Lustre: server umount lustre-OST0000 complete [ 3150.304622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.309832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.319300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.323311] Lustre: Skipped 1 previous similar message [ 3153.296337] LustreError: 190852:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3153.300543] LustreError: 190852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3153.336393] Lustre: server umount lustre-MDT0000 complete [ 3154.945892] LustreError: 186753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965869 with bad export cookie 11487177047926692607 [ 3154.953301] LustreError: 186753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3154.953405] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.043746] LustreError: 191055:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3155.046892] LustreError: 191055:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3155.102635] Lustre: server umount lustre-MDT0001 complete [ 3159.708265] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3161.207820] Key type lgssc unregistered [ 3161.392642] LNet: 191578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3161.398405] LNetError: 191578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3161.410789] LNet: Removed LNI 192.168.201.124@tcp [ 3161.947865] Key type .llcrypt unregistered [ 3161.950177] Key type ._llcrypt unregistered [ 3172.772925] Key type ._llcrypt registered [ 3172.774172] Key type .llcrypt registered [ 3172.815295] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3173.193829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.200725] alg: No test for adler32 (adler32-zlib) [ 3174.068566] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3174.165349] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3175.759186] Key type lgssc registered [ 3176.176038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3179.251751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3179.259061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.426324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.721143] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3184.696636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.837637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3186.389174] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3188.441523] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3189.722530] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3192.458796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3192.654795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.919356] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3196.961785] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3204.969750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3215.329584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.331114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.332067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.334382] Lustre: Skipped 1 previous similar message [ 3216.877080] LustreError: 195337:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3216.913506] Lustre: server umount lustre-OST0000 complete [ 3220.450252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.452371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.457066] Lustre: Skipped 1 previous similar message [ 3220.463574] Lustre: Skipped 2 previous similar messages [ 3224.475729] LustreError: 195539:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3224.480502] LustreError: 195539:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3224.520812] Lustre: server umount lustre-MDT0000 complete [ 3225.569336] LustreError: 192941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.573527] LustreError: 192941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3225.965316] LustreError: 192921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761965940 with bad export cookie 13223026733985767124 [ 3225.969110] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.969630] LustreError: 192921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3226.052412] LustreError: 195744:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3226.056345] LustreError: 195744:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3226.110167] Lustre: server umount lustre-MDT0001 complete [ 3230.215335] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3231.314834] Key type lgssc unregistered [ 3231.448571] LNet: 196268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.450925] LNetError: 196268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.462515] LNet: Removed LNI 192.168.201.124@tcp [ 3231.792656] Key type .llcrypt unregistered [ 3231.793580] Key type ._llcrypt unregistered [ 3240.199385] Key type ._llcrypt registered [ 3240.201260] Key type .llcrypt registered [ 3240.236099] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3240.793433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3240.803349] alg: No test for adler32 (adler32-zlib) [ 3241.684366] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3241.798427] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3243.407118] Key type lgssc registered [ 3243.778716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3247.076441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3247.083870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.286826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.704186] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3253.199626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3253.330580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3254.996713] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3257.234956] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3258.656709] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3261.108962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3261.302698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.642988] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3266.052559] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3273.574454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3283.938095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3283.940685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.944482] Lustre: Skipped 1 previous similar message [ 3283.944703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.947338] Lustre: Skipped 1 previous similar message [ 3286.126113] LustreError: 199880:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3286.162938] Lustre: server umount lustre-OST0000 complete [ 3289.060335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3289.060361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.061220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.062595] LustreError: Skipped 1 previous similar message [ 3289.068645] Lustre: Skipped 1 previous similar message [ 3293.628312] LustreError: 200082:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3293.630037] LustreError: 200082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3293.663113] Lustre: server umount lustre-MDT0000 complete [ 3294.176047] LustreError: 197483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3294.179921] LustreError: 197483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3294.888138] LustreError: 197464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966009 with bad export cookie 6377756769122927479 [ 3294.891143] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.891191] LustreError: 197464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3294.958484] LustreError: 200285:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3294.960373] LustreError: 200285:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3295.008507] Lustre: server umount lustre-MDT0001 complete [ 3298.618372] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3299.605614] Key type lgssc unregistered [ 3299.738418] LNet: 200809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3299.740824] LNetError: 200809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3299.749370] LNet: Removed LNI 192.168.201.124@tcp [ 3300.045245] Key type .llcrypt unregistered [ 3300.047150] Key type ._llcrypt unregistered [ 3307.546852] Key type ._llcrypt registered [ 3307.548305] Key type .llcrypt registered [ 3307.585027] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3307.955130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3307.964042] alg: No test for adler32 (adler32-zlib) [ 3308.825661] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3308.917451] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3310.503171] Key type lgssc registered [ 3310.866614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3313.851170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3313.857748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.015735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.293910] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3319.619931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3319.750762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.010948] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3322.696107] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3323.991994] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3326.457196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3326.573503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.438596] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3330.114741] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3339.881491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3343.289268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.326492] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3343.441190] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3345.459421] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3347.431316] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3347.970370] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3358.307316] Lustre: Permanently deactivating lustre-OST0001 [ 3358.313782] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3365.876721] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3368.032162] Lustre: Permanently reactivating lustre-OST0001 [ 3368.037346] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3368.039061] Lustre: Skipped 2 previous similar messages [ 3375.599088] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.603402] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3380.708162] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3381.423188] LustreError: 205442:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3381.443837] Lustre: server umount lustre-OST0001 complete [ 3385.828519] LustreError: 203607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.829729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.831532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3385.838554] LustreError: 203607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3385.851692] Lustre: Skipped 1 previous similar message [ 3390.946768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3390.950644] Lustre: Skipped 2 previous similar messages [ 3390.951562] LustreError: 203608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.063340] LustreError: 205446:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3396.072598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.074514] LustreError: 203607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.076002] Lustre: Skipped 1 previous similar message [ 3396.090491] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.087207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3397.112619] LustreError: 205648:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3397.116245] LustreError: 205648:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3397.171187] Lustre: server umount lustre-OST0000 complete [ 3416.159174] LustreError: 205653:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0000: unknown param osc.active=1 [ 3417.056274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3417.058484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.062297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.337520] LustreError: 205855:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3422.339218] LustreError: 205855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3422.370722] Lustre: server umount lustre-MDT0000 complete [ 3423.946865] LustreError: 204529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966138 with bad export cookie 7773955633007948592 [ 3423.952778] LustreError: 204529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3423.954492] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.117933] Lustre: server umount lustre-MDT0001 complete [ 3428.301170] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3429.563315] Key type lgssc unregistered [ 3429.697409] LNet: 206582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3429.699579] LNetError: 206582:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3429.709393] LNet: Removed LNI 192.168.201.124@tcp [ 3430.068047] Key type .llcrypt unregistered [ 3430.069101] Key type ._llcrypt unregistered [ 3440.540951] Key type ._llcrypt registered [ 3440.541970] Key type .llcrypt registered [ 3440.578328] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3446.463920] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3446.838784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3446.901668] alg: No test for adler32 (adler32-zlib) [ 3447.760461] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3447.846450] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3449.423132] Key type lgssc registered [ 3449.780878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3452.537992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.501470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3456.380896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3458.473220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.968355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.986955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3460.991016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.083414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3462.095621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3462.132750] Lustre: lustre-MDT0000: new disk, initializing [ 3462.169161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.176713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3463.480365] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3467.744894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.772520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3467.803885] Lustre: 210481:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3467.827256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3467.828905] Lustre: Skipped 1 previous similar message [ 3467.867201] Lustre: lustre-MDT0001: new disk, initializing [ 3467.900947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.913273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3467.916706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3469.142554] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3472.187821] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3473.391126] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3475.666109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3475.691802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3475.789402] Lustre: lustre-OST0000: new disk, initializing [ 3475.791270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3475.825141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.839979] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3480.847858] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3482.225361] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3483.148163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3483.155461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3483.206727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3483.356215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3484.595951] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3484.654326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3488.226264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3488.227476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.228362] LustreError: Skipped 1 previous similar message [ 3488.233929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.237962] Lustre: Skipped 1 previous similar message [ 3491.378193] LustreError: 213385:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3491.423326] Lustre: server umount lustre-OST0000 complete [ 3493.346957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.347894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3493.353967] Lustre: Skipped 2 previous similar messages [ 3493.359596] Lustre: Skipped 1 previous similar message [ 3498.464363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.467477] Lustre: Skipped 1 previous similar message [ 3498.816586] LustreError: 213586:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3498.818283] LustreError: 213586:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3498.848733] Lustre: server umount lustre-MDT0000 complete [ 3500.036793] LustreError: 211256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966214 with bad export cookie 7024734506781006450 [ 3500.042152] LustreError: 211256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3500.042614] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.124832] LustreError: 213788:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3500.127435] LustreError: 213788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3500.190091] Lustre: server umount lustre-MDT0001 complete [ 3502.834143] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 23:03:37 (1761966217) [ 3505.107985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.331277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.746313] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3509.704799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3509.817365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3510.999755] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3512.834030] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3513.999396] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3516.117432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3517.931424] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3519.581471] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.685599] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3540.186069] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3546.729387] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3557.471594] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3572.505342] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3572.507512] Lustre: Skipped 1 previous similar message [ 3590.667455] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3590.669751] Lustre: Skipped 1 previous similar message [ 3622.913852] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3622.917403] Lustre: Skipped 3 previous similar messages [ 3690.706893] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3690.709359] Lustre: Skipped 7 previous similar messages [ 3709.923804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.923836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.925899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3709.931399] Lustre: Skipped 1 previous similar message [ 3709.941813] LustreError: Skipped 1 previous similar message [ 3713.458138] LustreError: 217961:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3713.462186] LustreError: 217961:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3713.515233] Lustre: server umount lustre-OST0000 complete [ 3715.042291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.043912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.049352] Lustre: Skipped 1 previous similar message [ 3715.055588] Lustre: Skipped 2 previous similar messages [ 3725.280951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.284423] Lustre: Skipped 3 previous similar messages [ 3729.375197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3729.431451] LustreError: 218164:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3729.435167] LustreError: 218164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3729.471655] Lustre: server umount lustre-MDT0000 complete [ 3730.400664] LustreError: 217818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.410264] LustreError: 217818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3731.135442] LustreError: 214437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966445 with bad export cookie 7024734506781007325 [ 3731.140898] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.289440] Lustre: server umount lustre-MDT0001 complete [ 3735.509118] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3736.553104] Key type lgssc unregistered [ 3736.680448] LNet: 218890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3736.684894] LNetError: 218890:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3736.693292] LNet: Removed LNI 192.168.201.124@tcp [ 3736.979532] Key type .llcrypt unregistered [ 3736.981498] Key type ._llcrypt unregistered [ 3747.560592] Key type ._llcrypt registered [ 3747.562142] Key type .llcrypt registered [ 3747.603601] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3748.022394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3748.071476] alg: No test for adler32 (adler32-zlib) [ 3748.963489] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3749.076744] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3750.671220] Key type lgssc registered [ 3751.358489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3754.705666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3754.712445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.952988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.500569] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3760.801661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3760.988972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.655864] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3764.777107] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3766.407211] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3769.645693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3769.876814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.548674] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_default_debug -1 all [ 3774.632340] Lustre: DEBUG MARKER: oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3791.330142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.330196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3791.331292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.334524] Lustre: Skipped 1 previous similar message [ 3796.450377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3796.455892] Lustre: Skipped 2 previous similar messages [ 3797.420075] LustreError: 222647:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3797.451967] Lustre: server umount lustre-OST0000 complete [ 3801.569521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.575552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.577875] Lustre: Skipped 1 previous similar message [ 3806.688701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.690349] Lustre: Skipped 1 previous similar message [ 3811.808122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.809784] Lustre: Skipped 1 previous similar message [ 3813.343129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3813.375706] LustreError: 222852:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3813.377449] LustreError: 222852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3813.399274] Lustre: server umount lustre-MDT0000 complete [ 3814.557508] LustreError: 220232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761966529 with bad export cookie 16554541930536358581 [ 3814.560988] LustreError: 220232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3814.561216] LustreError: MGC192.168.201.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.626493] LustreError: 223057:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3814.628476] LustreError: 223057:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3814.677430] Lustre: server umount lustre-MDT0001 complete [ 3817.900140] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3818.941615] Key type lgssc unregistered [ 3819.079393] LNet: 223580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3819.082869] LNetError: 223580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3819.093426] LNet: Removed LNI 192.168.201.124@tcp [ 3819.394129] Key type .llcrypt unregistered [ 3819.395112] Key type ._llcrypt unregistered [ 3827.873647] Key type ._llcrypt registered [ 3827.874773] Key type .llcrypt registered [ 3827.904763] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3828.005276] Key type .llcrypt unregistered [ 3828.006393] Key type ._llcrypt unregistered [ 3837.366737] Key type ._llcrypt registered [ 3837.367989] Key type .llcrypt registered [ 3837.403688] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing set_hostid [ 3844.870609] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing load_modules_local [ 3845.258701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3845.269109] alg: No test for adler32 (adler32-zlib) [ 3846.121348] Lustre: Lustre: Build Version: 2.16.59_55_g8497249 [ 3846.210226] LNet: Added LNI 192.168.201.124@tcp [8/256/0/180] [ 3847.791302] Key type lgssc registered [ 3848.353753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3851.978398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3854.187154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3856.320068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3858.548133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3859.071512] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3714 sec ========== 23:09:34 (1761966574) [ 3859.600277] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:09:34 (1761966574) === [ 3860.743503] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:09:35 (1761966575) === [ 3875.797039] Lustre: DEBUG MARKER: oleg124-server.virtnet: executing unload_modules_local [ 3876.972547] Key type lgssc unregistered [ 3877.130758] LNet: 229630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3877.138764] LNetError: 229630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3877.149449] LNet: Removed LNI 192.168.201.124@tcp [ 3877.600607] Key type .llcrypt unregistered [ 3877.602498] Key type ._llcrypt unregistered