[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490475129 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007538] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.020693] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024686] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028122] *** VALIDATE tmpfs *** [ 0.030029] *** VALIDATE proc *** [ 0.031000] *** VALIDATE cgroup *** [ 0.031013] *** VALIDATE cgroup2 *** [ 0.033272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039295] debug: unmapping init [mem 0xffffffff98459000-0xffffffff98460fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042958] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044036] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046017] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051312] rcu: Hierarchical SRCU implementation. [ 0.053446] smp: Bringing up secondary CPUs ... [ 0.054406] x86: Booting SMP configuration: [ 0.055021] .... node #0, CPUs: #1 #2 #3 [ 0.067261] smp: Brought up 1 node, 4 CPUs [ 0.069012] smpboot: Max logical packages: 1 [ 0.070011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155067] node 0 deferred pages initialised in 82ms [ 0.158210] devtmpfs: initialized [ 0.159225] x86/mm: Memory block size: 128MB [ 0.162026] gcov: version magic: 0x41383552 [ 0.164246] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166256] pinctrl core: initialized pinctrl subsystem [ 0.167181] [ 0.167786] ************************************************************* [ 0.168013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169015] ** ** [ 0.170013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171018] ** ** [ 0.172015] ** This means that this kernel is built to expose internal ** [ 0.173012] ** IOMMU data structures, which may compromise security on ** [ 0.174010] ** your system. ** [ 0.175009] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.177010] ** kernel, report this immediately to your vendor! ** [ 0.178010] ** ** [ 0.179011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180013] ************************************************************* [ 0.181595] NET: Registered protocol family 16 [ 0.182437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185444] cpuidle: using governor menu [ 0.187176] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191323] PCI: Using configuration type 1 for base access [ 0.193121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205132] cryptd: max_cpu_qlen set to 1000 [ 0.206256] ACPI: Added _OSI(Module Device) [ 0.207010] ACPI: Added _OSI(Processor Device) [ 0.207967] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211178] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213392] ACPI: Interpreter enabled [ 0.214050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214944] ACPI: Using IOAPIC for interrupt routing [ 0.215101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216270] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225283] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232311] acpiphp: Slot [2] registered [ 0.234202] acpiphp: Slot [5] registered [ 0.236103] acpiphp: Slot [6] registered [ 0.237107] acpiphp: Slot [7] registered [ 0.240132] acpiphp: Slot [8] registered [ 0.241094] acpiphp: Slot [9] registered [ 0.243086] acpiphp: Slot [10] registered [ 0.244120] acpiphp: Slot [3] registered [ 0.245085] acpiphp: Slot [4] registered [ 0.247092] acpiphp: Slot [11] registered [ 0.248084] acpiphp: Slot [12] registered [ 0.250106] acpiphp: Slot [13] registered [ 0.251082] acpiphp: Slot [14] registered [ 0.253125] acpiphp: Slot [15] registered [ 0.254080] acpiphp: Slot [16] registered [ 0.256092] acpiphp: Slot [17] registered [ 0.257089] acpiphp: Slot [18] registered [ 0.258084] acpiphp: Slot [19] registered [ 0.260083] acpiphp: Slot [20] registered [ 0.260993] acpiphp: Slot [21] registered [ 0.261083] acpiphp: Slot [22] registered [ 0.261802] acpiphp: Slot [23] registered [ 0.263047] acpiphp: Slot [24] registered [ 0.263900] acpiphp: Slot [25] registered [ 0.264070] acpiphp: Slot [26] registered [ 0.266069] acpiphp: Slot [27] registered [ 0.267071] acpiphp: Slot [28] registered [ 0.268075] acpiphp: Slot [29] registered [ 0.269075] acpiphp: Slot [30] registered [ 0.271084] acpiphp: Slot [31] registered [ 0.276272] PCI host bridge to bus 0000:00 [ 0.279021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.282022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290018] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.293037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.294141] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.296565] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.300104] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.308724] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313519] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327544] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332046] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.335043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339566] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.361015] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.366793] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375948] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.396015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414015] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.423203] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.430011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454017] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.463276] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.469012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.474013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.486015] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.494818] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.506030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.518017] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.524771] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.530015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.535000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.542015] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.551678] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.556014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.562013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.574014] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.586290] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.589372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.591987] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.594342] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.596149] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600112] iommu: Default domain type: Passthrough [ 0.603412] SCSI subsystem initialized [ 0.604103] ACPI: bus type USB registered [ 0.605086] usbcore: registered new interface driver usbfs [ 0.607061] usbcore: registered new interface driver hub [ 0.609097] usbcore: registered new device driver usb [ 0.611152] pps_core: LinuxPPS API ver. 1 registered [ 0.613008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.616063] PTP clock support registered [ 0.618053] EDAC MC: Ver: 3.0.0 [ 0.620132] PCI: Using ACPI for IRQ routing [ 0.621788] NetLabel: Initializing [ 0.623008] NetLabel: domain hash size = 128 [ 0.624008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626079] NetLabel: unlabeled traffic allowed by default [ 0.628100] vgaarb: loaded [ 0.629377] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.631013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.640733] clocksource: Switched to clocksource kvm-clock [ 0.745983] VFS: Disk quotas dquot_6.6.0 [ 0.747027] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.749655] *** VALIDATE ramfs *** [ 0.751246] *** VALIDATE hugetlbfs *** [ 0.753049] pnp: PnP ACPI init [ 0.755537] pnp: PnP ACPI: found 6 devices [ 0.774310] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.776915] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.779093] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.781274] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.783535] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.785473] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.787806] NET: Registered protocol family 2 [ 0.790222] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.793634] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.796840] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.802439] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.805024] TCP: Hash tables configured (established 65536 bind 65536) [ 0.807722] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.809753] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.812336] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814986] NET: Registered protocol family 1 [ 0.816807] RPC: Registered named UNIX socket transport module. [ 0.819312] RPC: Registered udp transport module. [ 0.821083] RPC: Registered tcp transport module. [ 0.823297] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.826092] NET: Registered protocol family 44 [ 0.828456] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.831080] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.833634] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.836313] PCI: CLS 0 bytes, default 64 [ 0.838243] Unpacking initramfs... [ 2.318305] debug: unmapping init [mem 0xffff8bedfcc54000-0xffff8bedfffbffff] [ 2.325264] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.327974] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.331442] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.817266] Initialise system trusted keyrings [ 2.819082] Key type blacklist registered [ 2.820878] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.830041] zbud: loaded [ 2.833900] *** VALIDATE nfs *** [ 2.835588] *** VALIDATE nfs4 *** [ 2.837586] pstore: using deflate compression [ 2.846493] Platform Keyring initialized [ 2.995891] NET: Registered protocol family 38 [ 2.997736] Key type asymmetric registered [ 2.999205] Asymmetric key parser 'x509' registered [ 3.000800] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.003441] io scheduler mq-deadline registered [ 3.004866] io scheduler kyber registered [ 3.006464] io scheduler bfq registered [ 3.008400] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010983] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.013540] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.017831] ACPI: Power Button [PWRF] [ 3.112699] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.199480] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.387176] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.475873] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.664105] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.696421] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.731097] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.737141] Non-volatile memory driver v1.3 [ 3.738986] Linux agpgart interface v0.103 [ 3.769470] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.773490] vda: detected capacity change from 0 to 68505600 [ 3.789311] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.792367] vdb: detected capacity change from 0 to 1073741824 [ 3.807830] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.811059] vdc: detected capacity change from 0 to 2621440000 [ 3.826478] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.831546] vdd: detected capacity change from 0 to 2621440000 [ 3.852777] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.857375] vde: detected capacity change from 0 to 4294967296 [ 3.874494] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.877930] vdf: detected capacity change from 0 to 4294967296 [ 3.884455] libphy: Fixed MDIO Bus: probed [ 3.890224] usbcore: registered new interface driver usbserial_generic [ 3.894058] usbserial: USB Serial support registered for generic [ 3.897322] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.903045] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.905656] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.908711] mousedev: PS/2 mouse device common for all mice [ 3.912051] rtc_cmos 00:05: RTC can wake from S4 [ 3.914960] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.920544] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.925566] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.927078] rtc_cmos 00:05: registered as rtc0 [ 3.932714] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.936170] intel_pstate: CPU model not supported [ 3.939063] hid: raw HID events driver (C) Jiri Kosina [ 3.941179] usbcore: registered new interface driver usbhid [ 3.943312] usbhid: USB HID core driver [ 3.944958] drop_monitor: Initializing network drop monitor service [ 3.947309] Initializing XFRM netlink socket [ 3.949375] NET: Registered protocol family 10 [ 3.952072] Segment Routing with IPv6 [ 3.953579] NET: Registered protocol family 17 [ 3.955457] mpls_gso: MPLS GSO support [ 3.960658] RAS: Correctable Errors collector initialized. [ 3.962527] AVX version of gcm_enc/dec engaged. [ 3.963913] AES CTR mode by8 optimization enabled [ 4.061182] sched_clock: Marking stable (4061097185, 0)->(5011249000, -950151815) [ 4.064167] registered taskstats version 1 [ 4.065932] Loading compiled-in X.509 certificates [ 4.067808] zswap: loaded using pool lzo/zbud [ 4.099221] Key type big_key registered [ 4.113303] Key type encrypted registered [ 4.115132] ima: No TPM chip found, activating TPM-bypass! [ 4.117422] ima: Allocated hash algorithm: sha1 [ 4.119539] ima: No architecture policies found [ 4.121473] evm: Initialising EVM extended attributes: [ 4.123373] evm: security.selinux [ 4.124653] evm: security.ima [ 4.125735] evm: security.capability [ 4.127114] evm: HMAC attrs: 0x1 [ 4.129585] rtc_cmos 00:05: setting system clock to 2025-10-23 16:40:08 UTC (1761237608) [ 4.135756] debug: unmapping init [mem 0xffffffff99403000-0xffffffff995fffff] [ 4.139307] debug: unmapping init [mem 0xffffffff98182000-0xffffffff98458fff] [ 4.148112] Write protecting the kernel read-only data: 28672k [ 4.151561] debug: unmapping init [mem 0xffffffff96803000-0xffffffff969fffff] [ 4.155160] debug: unmapping init [mem 0xffffffff97114000-0xffffffff971fffff] [ 4.202599] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.212879] systemd[1]: Detected virtualization kvm. [ 4.215202] systemd[1]: Detected architecture x86-64. [ 4.217407] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.246791] systemd[1]: No hostname configured. [ 4.248549] systemd[1]: Set hostname to . [ 4.250932] random: systemd: uninitialized urandom read (16 bytes read) [ 4.255062] systemd[1]: Initializing machine ID from random generator. [ 4.284809] random: ln: uninitialized urandom read (6 bytes read) [ 4.384406] random: systemd: uninitialized urandom read (16 bytes read) [ 4.386935] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.391127] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.395355] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.098950] device-mapper: uevent: version 1.0.3 [ 5.101618] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 5.878338] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.957908] virtio_net virtio0 ens2: renamed from eth0 [ 6.019736] scsi host0: ata_piix [ 6.030196] scsi host1: ata_piix [ 6.034695] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.037489] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.632207] random: crng init done [ 10.633684] random: 7 urandom warning(s) missed due to ratelimiting [ 10.647545] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.428224] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.077082] printk: systemd: 22 output lines suppressed due to ratelimiting [ 13.376488] SELinux: Disabled at runtime. [ 13.438337] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.447940] systemd[1]: Detected virtualization kvm. [ 13.450440] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.107528] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.119475] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.123623] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.130901] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.138276] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.154343] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.161321] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 14.398371] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.801607] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.096213] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.175830] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.319389] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.337524] EDAC sbridge: Ver: 1.1.2 [ 17.411547] Key type dns_resolver registered [ 17.798575] NFS: Registering the id_resolver key type [ 17.801661] Key type id_resolver registered [ 17.803190] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 47.317626] libcfs: loading out-of-tree module taints kernel. [ 47.332256] Key type ._llcrypt registered [ 47.333932] Key type .llcrypt registered [ 47.390461] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 60.909395] hrtimer: interrupt took 4385284 ns [ 61.088683] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 62.639706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.651299] alg: No test for adler32 (adler32-zlib) [ 64.242354] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 65.093826] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 66.792168] Key type lgssc registered [ 68.592157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.294859] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.257558] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.489879] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.953026] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.215408] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.669708] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 148.579115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.681941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.718330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.100375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.170081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.290475] Lustre: lustre-MDT0000: new disk, initializing [ 150.415631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.446301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.618396] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.129245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.260955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 168.403520] Lustre: 6517:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 168.465219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 168.478191] Lustre: Skipped 1 previous similar message [ 168.627239] Lustre: lustre-MDT0001: new disk, initializing [ 168.725995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 168.774449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 168.786515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.876973] LustreError: 7466:0:(llog_cat.c:511:llog_cat_current_log()) lustre-MDT0001-osd: next log does not exist, catalog [0x240000400:0x1:0x0] rc=-5 [ 168.885602] LustreError: 7466:0:(update_trans.c:965:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -5 [ 168.898706] LustreError: 7466:0:(mdt_handler.c:8387:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 173.166043] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.412384] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 188.120475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.206813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.481685] Lustre: lustre-OST0000: new disk, initializing [ 188.484170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 188.585169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.500457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.517095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.615316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.457266] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.248599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.370301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 211.555412] Lustre: lustre-OST0001: new disk, initializing [ 211.557848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.682366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.178646] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.152744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.176634] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.213393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.528547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.801363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.032752] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.506788] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 258.379969] Lustre: DEBUG MARKER: Client: 2.16.59.38 [ 260.837824] Lustre: DEBUG MARKER: MDS: 2.16.59.38 [ 263.376656] Lustre: DEBUG MARKER: OSS: 2.16.59.38 [ 264.812852] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 23 12:44:29 EDT 2025 [ 284.560588] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 286.096255] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.024074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.027205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.063072] Lustre: Skipped 2 previous similar messages [ 301.068396] Lustre: Skipped 3 previous similar messages [ 306.145616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.153536] Lustre: Skipped 3 previous similar messages [ 306.640388] LustreError: 12225:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 306.712440] Lustre: server umount lustre-MDT0000 complete [ 310.805762] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237915 with bad export cookie 12142085485978434112 [ 310.812060] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.822904] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 311.265813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.267600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.289547] Lustre: Skipped 2 previous similar messages [ 311.295118] Lustre: Skipped 1 previous similar message [ 315.298988] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.303474] Lustre: Skipped 1 previous similar message [ 317.180590] LustreError: 12427:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 317.183384] LustreError: 12427:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 317.411614] Lustre: server umount lustre-MDT0001 complete [ 321.105098] LustreError: 12628:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 321.107729] LustreError: 12628:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 321.205911] Lustre: server umount lustre-OST0000 complete [ 324.338334] LustreError: 12829:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 324.343857] LustreError: 12829:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 324.609159] Lustre: server umount lustre-OST0001 complete [ 330.322566] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 340.418489] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 350.931582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.474831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.246949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 370.849534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 376.430185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 376.500094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.705134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 376.729582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 376.773598] Lustre: lustre-MDT0000: new disk, initializing [ 376.844154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.862468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 380.431481] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 390.822953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 390.884994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 390.981971] Lustre: 15018:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.055269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.057888] Lustre: Skipped 1 previous similar message [ 391.117119] Lustre: lustre-MDT0001: new disk, initializing [ 391.214887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 391.251129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 391.265052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.338995] Lustre: 15816:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 391.356026] Lustre: 15816:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 391.362945] Lustre: 15816:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 391.373016] Lustre: 15816:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 391.392421] Lustre: 15816:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 391.402584] Lustre: 15816:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 394.819096] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 402.682970] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 406.603380] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 413.192380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 413.291357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 413.653199] Lustre: lustre-OST0000: new disk, initializing [ 413.661746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 413.747349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.278372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 415.301801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.437096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 419.620716] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 426.212806] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 429.967971] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 430.262659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.375183] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.594607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.312568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.318540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.337799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.347880] Lustre: Skipped 1 previous similar message [ 442.633171] LustreError: 17958:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 442.636144] LustreError: 17958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 442.823392] Lustre: server umount lustre-OST0000 complete [ 446.432792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.554282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.556994] Lustre: Skipped 3 previous similar messages [ 452.384165] LustreError: 18160:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 452.391767] LustreError: 18160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 452.481779] Lustre: server umount lustre-MDT0000 complete [ 456.672289] LustreError: 15817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.685925] LustreError: 15817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 456.847415] LustreError: 15800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238061 with bad export cookie 12142085485978435715 [ 456.848271] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.874583] LustreError: 15800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 457.396949] Lustre: server umount lustre-MDT0001 complete [ 461.401903] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:47:45 (1761238065) [ 469.415872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.262892] LustreError: 18887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.398736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.633974] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 475.618194] LustreError: 18887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.714184] LustreError: 18887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.682625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 486.800690] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 491.876976] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.674380] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 502.417774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 508.016172] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 513.008097] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 528.356401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.361258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.368644] Lustre: Skipped 1 previous similar message [ 528.382133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.385968] Lustre: Skipped 1 previous similar message [ 531.466327] LustreError: 21153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 531.478369] LustreError: 21153:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 531.593963] Lustre: server umount lustre-OST0000 complete [ 541.569611] Lustre: server umount lustre-MDT0000 complete [ 544.669278] LustreError: 18867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238149 with bad export cookie 12142085485978436632 [ 544.681219] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.693477] LustreError: 18867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 545.132648] Lustre: server umount lustre-MDT0001 complete [ 554.873829] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 557.632227] Key type lgssc unregistered [ 557.995758] LNet: 22090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.999426] LNetError: 22090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.033597] LNet: Removed LNI 192.168.201.105@tcp [ 559.153200] Key type .llcrypt unregistered [ 559.158637] Key type ._llcrypt unregistered [ 578.273227] Key type ._llcrypt registered [ 578.274899] Key type .llcrypt registered [ 578.388562] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 579.070500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 579.175795] alg: No test for adler32 (adler32-zlib) [ 580.207550] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 580.372579] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 582.047174] Key type lgssc registered [ 582.945842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 588.969872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 588.980472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.377366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.747808] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 600.545168] LustreError: 23465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.555380] LustreError: 23465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 600.664742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 600.945386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 603.988171] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 607.872729] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 610.442142] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 614.905453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 615.162534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.172897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 618.672292] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 622.533721] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 636.900854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.902319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.912120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.921253] Lustre: Skipped 1 previous similar message [ 640.764920] LustreError: 26152:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 640.864810] Lustre: server umount lustre-OST0000 complete [ 647.138385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 647.139813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.151449] LustreError: Skipped 1 previous similar message [ 647.157488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.161737] Lustre: Skipped 1 previous similar message [ 647.174285] Lustre: Skipped 2 previous similar messages [ 652.258201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.261130] Lustre: Skipped 1 previous similar message [ 657.379447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.383749] Lustre: Skipped 1 previous similar message [ 658.399870] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 658.455164] LustreError: 26353:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 658.460018] LustreError: 26353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 658.519820] Lustre: server umount lustre-MDT0000 complete [ 660.594282] LustreError: 23445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238264 with bad export cookie 4818603328983704057 [ 660.599668] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.601438] LustreError: 23445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 660.695164] LustreError: 26556:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 660.698703] LustreError: 26556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 660.827170] Lustre: server umount lustre-MDT0001 complete [ 667.093712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 669.143618] Key type lgssc unregistered [ 669.377886] LNet: 27089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.381572] LNetError: 27089:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.393801] LNet: Removed LNI 192.168.201.105@tcp [ 670.031131] Key type .llcrypt unregistered [ 670.038610] Key type ._llcrypt unregistered [ 685.896053] Key type ._llcrypt registered [ 685.897497] Key type .llcrypt registered [ 685.966624] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 686.580451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.595680] alg: No test for adler32 (adler32-zlib) [ 687.619367] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 687.852254] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 689.479176] Key type lgssc registered [ 690.351196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.224271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.244220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.557106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.883779] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 704.816818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 705.099522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 708.267306] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 712.510481] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.269981] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 726.171921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 726.566265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.778839] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 735.145160] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.935729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 752.098286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.098814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 752.100980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 752.113783] Lustre: Skipped 1 previous similar message [ 752.130452] Lustre: Skipped 1 previous similar message [ 756.534189] LustreError: 31076:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 756.686311] Lustre: server umount lustre-OST0000 complete [ 761.313938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.325663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.348613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.337687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.357225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.367819] Lustre: Skipped 1 previous similar message [ 766.286463] LustreError: 31277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 766.296555] LustreError: 31277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 766.395604] Lustre: server umount lustre-MDT0000 complete [ 767.459865] LustreError: 30630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.503046] LustreError: 30630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 770.206607] LustreError: 28444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238374 with bad export cookie 18432814695414287204 [ 770.210935] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.227736] LustreError: 28444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 770.327024] LustreError: 31479:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 770.336946] LustreError: 31479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 770.632840] Lustre: server umount lustre-MDT0001 complete [ 779.907761] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 782.586196] Key type lgssc unregistered [ 782.977426] LNet: 32012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.988438] LNetError: 32012:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.005881] LNet: Removed LNI 192.168.201.105@tcp [ 783.807306] Key type .llcrypt unregistered [ 783.808981] Key type ._llcrypt unregistered [ 803.043818] Key type ._llcrypt registered [ 803.045413] Key type .llcrypt registered [ 803.134205] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 803.909734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.932420] alg: No test for adler32 (adler32-zlib) [ 804.920930] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 805.118982] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 806.745164] Key type lgssc registered [ 807.800536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 815.201790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 815.216251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.581041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.887638] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 825.839596] LustreError: 33384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 825.868700] LustreError: 33384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 828.224978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 828.768058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.706875] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 837.660810] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 841.109550] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 847.637185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 847.919577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.192030] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 859.995869] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 861.247115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 873.440379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 873.443302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.468197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 873.960283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 873.970846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.986415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 876.513780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.703897] LustreError: 35633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 877.901631] Lustre: server umount lustre-OST0000 complete [ 881.635823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.644141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.650769] Lustre: Skipped 1 previous similar message [ 881.653952] Lustre: Skipped 1 previous similar message [ 886.753556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 886.765617] Lustre: Skipped 1 previous similar message [ 886.872629] LustreError: 35834:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 886.879554] LustreError: 35834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 886.990224] Lustre: server umount lustre-MDT0000 complete [ 890.377757] LustreError: 33366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238494 with bad export cookie 212609682335768865 [ 890.388322] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.498676] LustreError: 36039:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 890.505399] LustreError: 36039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 890.758928] Lustre: server umount lustre-MDT0001 complete [ 900.819106] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 903.618748] Key type lgssc unregistered [ 903.927616] LNet: 36572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.942104] LNetError: 36572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.974338] LNet: Removed LNI 192.168.201.105@tcp [ 904.817313] Key type .llcrypt unregistered [ 904.819555] Key type ._llcrypt unregistered [ 924.740412] Key type ._llcrypt registered [ 924.742703] Key type .llcrypt registered [ 924.843565] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 925.665711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.727067] alg: No test for adler32 (adler32-zlib) [ 926.797772] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 927.047294] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 928.719598] Key type lgssc registered [ 929.635550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.029623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.049263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.483826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.857610] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 946.664223] LustreError: 37945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.711100] LustreError: 37945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 949.597256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 950.207587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 954.056739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 958.466359] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 961.266578] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 966.003864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 966.237759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.851304] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 973.709929] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 982.649070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 990.940951] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 991.201798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.210200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.249618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.253490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.263196] Lustre: Skipped 1 previous similar message [ 993.274408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.023908] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 996.029364] Lustre: Skipped 1 previous similar message [ 996.630136] LustreError: 40215:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 996.942347] Lustre: server umount lustre-OST0000 complete [ 1016.802637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1016.805858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.813370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1020.767408] LustreError: 40593:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1020.770393] LustreError: 40593:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1020.823853] Lustre: server umount lustre-MDT0000 complete [ 1022.948740] LustreError: 37941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.967873] LustreError: 37941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1023.893735] LustreError: 37926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238628 with bad export cookie 16065614796550623233 [ 1023.901685] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.912145] LustreError: 37926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1038.303645] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1038.378710] LustreError: 40796:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1038.384023] LustreError: 40796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1038.567240] Lustre: server umount lustre-MDT0001 complete [ 1045.795520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1048.064676] Key type lgssc unregistered [ 1048.335065] LNet: 41331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.343109] LNetError: 41331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.354792] LNet: Removed LNI 192.168.201.105@tcp [ 1049.147604] Key type .llcrypt unregistered [ 1049.149330] Key type ._llcrypt unregistered [ 1066.384279] Key type ._llcrypt registered [ 1066.385996] Key type .llcrypt registered [ 1066.465289] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1067.327290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.341429] alg: No test for adler32 (adler32-zlib) [ 1068.360875] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1068.537313] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1070.175172] Key type lgssc registered [ 1070.901350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.770587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.788670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.185889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.656747] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1088.206542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1088.451654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.288776] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1095.574927] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1098.719655] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1104.230974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1104.541788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.949317] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1113.413672] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1124.011987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1129.443959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.445954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1129.447052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.449468] Lustre: Skipped 1 previous similar message [ 1130.185270] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 1130.189337] Lustre: Skipped 2 previous similar messages [ 1131.168345] LustreError: 44971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1131.238052] Lustre: server umount lustre-MDT0000 complete [ 1133.854945] LustreError: 42684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238738 with bad export cookie 9758635227887679462 [ 1133.855734] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.862390] LustreError: 42684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1134.563636] LustreError: 43818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.564814] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.572467] LustreError: 43818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1134.586902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1135.282171] LustreError: 42698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.966466] LustreError: 45174:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1135.969382] LustreError: 45174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1136.093043] Lustre: server umount lustre-MDT0001 complete [ 1147.641818] LustreError: 42683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.5@tcp arrived at 1761238752 with bad export cookie 9758635227887680134 [ 1147.650259] LustreError: 42683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1155.642381] LustreError: 45396:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1155.648592] LustreError: 45396:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1155.869398] Lustre: server umount lustre-OST0000 complete [ 1167.344860] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1169.313894] Key type lgssc unregistered [ 1169.563729] LNet: 46233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.568296] LNetError: 46233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.582155] LNet: Removed LNI 192.168.201.105@tcp [ 1170.194428] Key type .llcrypt unregistered [ 1170.196135] Key type ._llcrypt unregistered [ 1188.000892] Key type ._llcrypt registered [ 1188.002636] Key type .llcrypt registered [ 1188.093327] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1189.122059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.238917] alg: No test for adler32 (adler32-zlib) [ 1190.328872] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1190.526975] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1192.207190] Key type lgssc registered [ 1193.038410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1198.960078] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1198.971495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1214.495368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1219.615576] LustreError: 47580:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 1246.239274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.937402] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1253.509809] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1417.791748] LustreError: 48299:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1417.971569] Lustre: server umount lustre-OST0000 complete [ 1430.560308] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1433.158817] Key type lgssc unregistered [ 1433.456908] LNet: 49136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.466891] LNetError: 49136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.489539] LNet: Removed LNI 192.168.201.105@tcp [ 1434.273620] Key type .llcrypt unregistered [ 1434.275327] Key type ._llcrypt unregistered [ 1452.082896] Key type ._llcrypt registered [ 1452.084510] Key type .llcrypt registered [ 1452.174598] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1452.855430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1452.865786] alg: No test for adler32 (adler32-zlib) [ 1453.844142] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1454.024646] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1455.703186] Key type lgssc registered [ 1456.574749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1463.103170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1463.122363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.539385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.098150] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1474.023425] LustreError: 50511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.052905] LustreError: 50511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1476.352586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1476.712346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.876992] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1484.085410] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1486.953512] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1492.628827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1492.862084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.540575] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1502.538745] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1507.298294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.305050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.305471] LustreError: Skipped 1 previous similar message [ 1507.319186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.346942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1510.524509] LustreError: 52755:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1510.695755] Lustre: server umount lustre-OST0000 complete [ 1514.465967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.491239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.501279] Lustre: Skipped 2 previous similar messages [ 1519.588679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.591314] Lustre: Skipped 1 previous similar message [ 1520.497647] LustreError: 52959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1520.500714] LustreError: 52959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1520.571218] Lustre: server umount lustre-MDT0000 complete [ 1523.205316] LustreError: 50491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239127 with bad export cookie 1028250503101905802 [ 1523.211585] LustreError: 51226:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 1523.214938] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.218364] LustreError: 50491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1523.241888] LustreError: 51226:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 1523.306797] LustreError: 53160:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1523.309136] LustreError: 53160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1523.453257] Lustre: server umount lustre-MDT0001 complete [ 1530.239617] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1532.096785] Key type lgssc unregistered [ 1532.280740] LNet: 53694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.286930] LNetError: 53694:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.297084] LNet: Removed LNI 192.168.201.105@tcp [ 1532.922189] Key type .llcrypt unregistered [ 1532.923594] Key type ._llcrypt unregistered [ 1549.192306] Key type ._llcrypt registered [ 1549.194023] Key type .llcrypt registered [ 1549.281653] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1549.991527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.044359] alg: No test for adler32 (adler32-zlib) [ 1551.018406] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1551.171955] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1552.801183] Key type lgssc registered [ 1553.711687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.070721] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1560.082275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1575.647328] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1580.771205] LustreError: 55038:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 1607.455241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.393111] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1616.686375] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1623.054250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.486776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.565866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1626.988921] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1628.642378] LustreError: 55065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.685663] LustreError: 55065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1633.778509] LustreError: 55065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.140500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1634.398603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.402320] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1641.468163] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1644.431612] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1647.079355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.089120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.193869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.197265] Lustre: Skipped 1 previous similar message [ 1657.325503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1657.329624] Lustre: Skipped 1 previous similar message [ 1660.383160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1660.411781] LustreError: 57316:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1660.529984] Lustre: server umount lustre-OST0000 complete [ 1680.358450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1680.363582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.379172] Lustre: Skipped 1 previous similar message [ 1680.382143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1680.385862] Lustre: Skipped 1 previous similar message [ 1682.914699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.338360] LustreError: 57696:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1685.342297] LustreError: 57696:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1685.436672] Lustre: server umount lustre-MDT0000 complete [ 1687.008438] LustreError: 55966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.026234] LustreError: 55966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1688.119349] LustreError: 55045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239292 with bad export cookie 17882466269790113884 [ 1688.125256] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.126182] LustreError: 55045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1694.291540] LustreError: 57898:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1694.293558] LustreError: 57898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1694.437424] Lustre: server umount lustre-MDT0001 complete [ 1701.332615] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1703.259078] Key type lgssc unregistered [ 1703.460412] LNet: 58431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.464578] LNetError: 58431:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.484758] LNet: Removed LNI 192.168.201.105@tcp [ 1704.128135] Key type .llcrypt unregistered [ 1704.132348] Key type ._llcrypt unregistered [ 1721.474630] Key type ._llcrypt registered [ 1721.476249] Key type .llcrypt registered [ 1721.590581] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1722.282432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.359089] alg: No test for adler32 (adler32-zlib) [ 1723.401750] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1723.569968] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1725.223201] Key type lgssc registered [ 1726.410355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1732.648078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1732.659613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.001275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.015975] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1744.127894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1744.401517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.609242] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1751.157651] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1753.727790] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1758.489824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1758.786888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.856567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1763.238135] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1766.939712] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1784.806941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.809085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.809286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1784.820370] Lustre: Skipped 1 previous similar message [ 1787.719378] LustreError: 62072:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1787.847102] Lustre: server umount lustre-OST0000 complete [ 1790.433778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1790.444097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.463323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.469868] Lustre: Skipped 1 previous similar message [ 1795.040294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.041356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.513604] LustreError: 62273:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1796.520337] LustreError: 62273:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1796.609920] Lustre: server umount lustre-MDT0000 complete [ 1799.682235] LustreError: 59785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239404 with bad export cookie 3099916769713507952 [ 1799.686788] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.689211] LustreError: 59785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1799.833608] LustreError: 62476:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1799.838793] LustreError: 62476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1800.051982] Lustre: server umount lustre-MDT0001 complete [ 1809.027103] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1811.293681] Key type lgssc unregistered [ 1811.505979] LNet: 63008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.513268] LNetError: 63008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.530673] LNet: Removed LNI 192.168.201.105@tcp [ 1812.104100] Key type .llcrypt unregistered [ 1812.111763] Key type ._llcrypt unregistered [ 1839.087630] Key type ._llcrypt registered [ 1839.089672] Key type .llcrypt registered [ 1839.155074] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1839.832303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.891186] alg: No test for adler32 (adler32-zlib) [ 1840.835328] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1840.988274] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1842.639222] Key type lgssc registered [ 1843.700293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.218892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1850.238382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.691805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.780502] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1861.747884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1862.043451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1865.178607] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1870.032942] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1872.811074] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1878.228577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.376384] LustreError: 66272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1878.409572] LustreError: 66272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1878.578581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.190596] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1883.639843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1887.150610] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.416098] Lustre: Failing over lustre-MDT0000 [ 1891.615592] LustreError: 66918:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1891.695914] Lustre: server umount lustre-MDT0000 complete [ 1891.814271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.815235] LustreError: 64674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.819625] Lustre: Skipped 1 previous similar message [ 1891.824577] LustreError: 64674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1893.580292] LustreError: 64668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.615438] LustreError: 64668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1896.930767] LustreError: 66831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.942898] LustreError: 66831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1898.654457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.837947] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.969517] Lustre: *** cfs_fail_loc=135, val=0*** [ 1899.063698] LustreError: 67427:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1899.070190] LustreError: 67427:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -2 [ 1899.084581] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1899.084581] [ 1899.092194] LustreError: MGC192.168.201.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1899.110183] LustreError: 67415:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1899.114254] LustreError: 67415:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 1899.128765] LustreError: 67415:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1899.152987] Lustre: server umount lustre-MDT0000 complete [ 1899.162144] LustreError: 67415:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1902.059310] LustreError: 64673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.073347] LustreError: 64673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1905.200901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.367663] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.673857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.776597] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1908.919340] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1909.933401] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1910.763877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1910.798750] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1910.840929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1916.384211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.393247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.421156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1917.410709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1917.420350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.434288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.995372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.242815] LustreError: 68238:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1922.257287] LustreError: 68238:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1922.451126] Lustre: server umount lustre-OST0000 complete [ 1926.118315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.118801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.120599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.120606] Lustre: Skipped 1 previous similar message [ 1926.141749] Lustre: Skipped 1 previous similar message [ 1931.232639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.235334] Lustre: Skipped 1 previous similar message [ 1931.950278] LustreError: 68439:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1931.952837] LustreError: 68439:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1932.033974] Lustre: server umount lustre-MDT0000 complete [ 1935.064973] LustreError: 64654:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239539 with bad export cookie 5098719183009716093 [ 1935.071057] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.079656] LustreError: 64654:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1935.190090] LustreError: 68640:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1935.193198] LustreError: 68640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1935.405322] Lustre: server umount lustre-MDT0001 complete [ 1942.528908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1944.684901] Key type lgssc unregistered [ 1944.944491] LNet: 69172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.948659] LNetError: 69172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.965138] LNet: Removed LNI 192.168.201.105@tcp [ 1945.873509] Key type .llcrypt unregistered [ 1945.875953] Key type ._llcrypt unregistered [ 1967.003278] Key type ._llcrypt registered [ 1967.004970] Key type .llcrypt registered [ 1967.112923] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1967.880097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.986087] alg: No test for adler32 (adler32-zlib) [ 1968.938785] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1969.120723] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1970.739300] Key type lgssc registered [ 1971.528891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1977.165118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1977.188334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.552682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.154091] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1988.069202] LustreError: 70565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.083232] LustreError: 70565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1988.357814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.650756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.405559] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1994.806437] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.088468] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2001.596797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2001.880893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.932537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2005.333738] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2008.962103] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.869042] Lustre: Failing over lustre-MDT0000 [ 2017.987395] LustreError: 72814:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2018.065110] Lustre: server umount lustre-MDT0000 complete [ 2018.276702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.280712] LustreError: 70561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.282128] Lustre: Skipped 1 previous similar message [ 2018.297768] LustreError: 70561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2021.571942] LustreError: 70562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.685712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.803990] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.894486] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2024.968557] LustreError: 73322:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2024.973083] LustreError: 73322:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -9 [ 2024.978056] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2024.978056] [ 2024.984675] LustreError: MGC192.168.201.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2024.993844] LustreError: 73310:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2024.998651] LustreError: 73310:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -9 [ 2025.004445] LustreError: 73310:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2025.016256] LustreError: 70547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239629 with bad export cookie 10267193639636056882 [ 2025.029947] Lustre: server umount lustre-MDT0000 complete [ 2025.037243] LustreError: 73310:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2026.674644] LustreError: 71860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.688833] LustreError: 71860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2029.449111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.791753] LustreError: 72726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.802207] LustreError: 72726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2034.655952] LustreError: 69874:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000053de7f08 x1846793546524928/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2034.829112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.867947] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2036.910272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2037.398285] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2040.297710] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2040.309404] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2040.339750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2043.881264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.889765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.897625] Lustre: Skipped 1 previous similar message [ 2043.901835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.408333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.413703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.414339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.426464] Lustre: Skipped 1 previous similar message [ 2048.894300] LustreError: 74137:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2048.896847] LustreError: 74137:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2049.084694] Lustre: server umount lustre-OST0000 complete [ 2055.649086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.649509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.650628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.657026] Lustre: Skipped 1 previous similar message [ 2060.776186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.785022] Lustre: Skipped 2 previous similar messages [ 2065.889629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.893459] Lustre: Skipped 1 previous similar message [ 2066.399189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2066.482801] LustreError: 74339:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2066.491440] LustreError: 74339:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2066.584132] Lustre: server umount lustre-MDT0000 complete [ 2069.376539] LustreError: 70546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239673 with bad export cookie 10267193639636057036 [ 2069.388194] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.390030] LustreError: 70546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2069.408455] LustreError: Skipped 1 previous similar message [ 2069.488678] LustreError: 74542:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2069.497413] LustreError: 74542:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2069.667781] Lustre: server umount lustre-MDT0001 complete [ 2078.427262] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2081.689554] Key type lgssc unregistered [ 2082.115710] LNet: 75075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.120019] LNetError: 75075:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.151894] LNet: Removed LNI 192.168.201.105@tcp [ 2083.012836] Key type .llcrypt unregistered [ 2083.014103] Key type ._llcrypt unregistered [ 2095.584158] Key type ._llcrypt registered [ 2095.588614] Key type .llcrypt registered [ 2095.687364] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2096.032476] Key type .llcrypt unregistered [ 2096.034287] Key type ._llcrypt unregistered [ 2115.538648] Key type ._llcrypt registered [ 2115.540144] Key type .llcrypt registered [ 2115.631460] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2116.328551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2116.415886] alg: No test for adler32 (adler32-zlib) [ 2117.460649] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2117.639083] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2119.335190] Key type lgssc registered [ 2120.293884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.774929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.802812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.282993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.098572] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2138.597053] LustreError: 76937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.612473] LustreError: 76937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2139.579694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2139.887621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.566666] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2148.456254] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.892147] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2157.589583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2157.876601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.242666] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2168.217924] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2173.246193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2183.648432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.650672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.657568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.661223] Lustre: Skipped 1 previous similar message [ 2183.666754] Lustre: Skipped 1 previous similar message [ 2187.381122] LustreError: 79205:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2187.571088] Lustre: server umount lustre-OST0000 complete [ 2191.328192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.332318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.339583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.890489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.891740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.246636] LustreError: 79406:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2197.378803] LustreError: 79406:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2197.710896] Lustre: server umount lustre-MDT0000 complete [ 2199.008336] LustreError: 77650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.039347] LustreError: 77650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2201.369783] LustreError: 76918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239805 with bad export cookie 10739552563900480073 [ 2201.377716] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.380597] LustreError: 76918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2201.486413] LustreError: 79608:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2201.490225] LustreError: 79608:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2201.652708] Lustre: server umount lustre-MDT0001 complete [ 2210.740061] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2213.759913] Key type lgssc unregistered [ 2214.017959] LNet: 80140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.032594] LNetError: 80140:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.046945] LNet: Removed LNI 192.168.201.105@tcp [ 2214.983468] Key type .llcrypt unregistered [ 2214.986717] Key type ._llcrypt unregistered [ 2236.531636] Key type ._llcrypt registered [ 2236.533849] Key type .llcrypt registered [ 2236.661924] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2237.788508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2237.808743] alg: No test for adler32 (adler32-zlib) [ 2238.993944] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2239.244284] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2240.919228] Key type lgssc registered [ 2242.144673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2249.816768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2249.845330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.423071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.871755] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2260.967836] LustreError: 81514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.988296] LustreError: 81514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2262.604161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2262.831658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.454160] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2271.143746] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2274.668429] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2281.431925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2281.913518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.818187] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2294.483632] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2296.264405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2306.538591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.540668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.583902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2311.252161] LustreError: 83765:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2311.483245] Lustre: server umount lustre-OST0000 complete [ 2316.773605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.779331] Lustre: Skipped 1 previous similar message [ 2316.783321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.791724] Lustre: Skipped 2 previous similar messages [ 2321.311281] LustreError: 83966:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2321.319140] LustreError: 83966:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2321.420325] Lustre: server umount lustre-MDT0000 complete [ 2321.892545] LustreError: 81511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.911277] LustreError: 81511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2324.366246] LustreError: 81495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239928 with bad export cookie 4763931159736191755 [ 2324.371772] LustreError: 81495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2324.371925] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.515301] LustreError: 84168:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2324.518339] LustreError: 84168:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2324.761552] Lustre: server umount lustre-MDT0001 complete [ 2333.067487] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2335.885376] Key type lgssc unregistered [ 2336.123899] LNet: 84700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.138928] LNetError: 84700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.176249] LNet: Removed LNI 192.168.201.105@tcp [ 2337.314276] Key type .llcrypt unregistered [ 2337.316483] Key type ._llcrypt unregistered [ 2356.352977] Key type ._llcrypt registered [ 2356.355774] Key type .llcrypt registered [ 2356.494969] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2357.514989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.543212] alg: No test for adler32 (adler32-zlib) [ 2358.598341] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2358.811303] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2360.535188] Key type lgssc registered [ 2361.835746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2368.228656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.245826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.731237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.874408] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2379.885760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2380.227512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2383.678077] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2389.495364] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2392.799498] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2398.439735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2398.766853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.203956] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2407.040183] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2416.177876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2426.339576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2426.343308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.344852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.356354] Lustre: Skipped 1 previous similar message [ 2427.519171] LustreError: 88352:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2427.720726] Lustre: server umount lustre-OST0000 complete [ 2431.456417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.456501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2431.478520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.495660] Lustre: Skipped 2 previous similar messages [ 2436.580309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.422720] LustreError: 88552:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2437.431509] LustreError: 88552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2437.508253] Lustre: server umount lustre-MDT0000 complete [ 2440.834574] LustreError: 86054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240045 with bad export cookie 7720858922440033934 [ 2440.836503] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.841477] LustreError: 86054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2440.987930] LustreError: 88756:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2441.016354] LustreError: 88756:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2441.226203] Lustre: server umount lustre-MDT0001 complete [ 2449.273792] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2451.892740] Key type lgssc unregistered [ 2452.181190] LNet: 89287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.191914] LNetError: 89287:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.207205] LNet: Removed LNI 192.168.201.105@tcp [ 2452.829079] Key type .llcrypt unregistered [ 2452.831973] Key type ._llcrypt unregistered [ 2472.026201] Key type ._llcrypt registered [ 2472.027818] Key type .llcrypt registered [ 2472.146543] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2472.992038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.017040] alg: No test for adler32 (adler32-zlib) [ 2474.044698] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2474.196473] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2475.864311] Key type lgssc registered [ 2476.915748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.846097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2484.879054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2500.447312] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2505.569226] LustreError: 90632:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 2532.191204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.218976] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2542.392248] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2547.411264] LustreError: 91499:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2547.532462] Lustre: server umount lustre-OST0000 complete [ 2554.276991] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:22:38 (1761240158) [ 2559.984228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.398276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.548191] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2565.601056] LustreError: 92169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.616433] LustreError: 92169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2569.704264] LustreError: 92169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.906960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2570.196662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.397529] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2577.816614] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2581.312674] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2586.457478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2586.759608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.205733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2595.635918] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2605.304580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2615.778298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.781215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.792087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.795030] Lustre: Skipped 1 previous similar message [ 2620.700738] LustreError: 94596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2620.705235] LustreError: 94596:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2620.891572] Lustre: server umount lustre-OST0000 complete [ 2626.017971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.036141] Lustre: Skipped 1 previous similar message [ 2626.044802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.047452] Lustre: Skipped 1 previous similar message [ 2630.728132] LustreError: 94797:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2630.733507] LustreError: 94797:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2630.800892] Lustre: server umount lustre-MDT0000 complete [ 2631.136902] LustreError: 92164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.155371] LustreError: 92164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2634.531780] LustreError: 92151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240238 with bad export cookie 16135314281875932336 [ 2634.535046] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.556075] LustreError: 92151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2634.784896] LustreError: 95002:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2634.796126] LustreError: 95002:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2635.096435] Lustre: server umount lustre-MDT0001 complete [ 2643.958056] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2646.466832] Key type lgssc unregistered [ 2646.718516] LNet: 95534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.728352] LNetError: 95534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.742928] LNet: Removed LNI 192.168.201.105@tcp [ 2647.650631] Key type .llcrypt unregistered [ 2647.653365] Key type ._llcrypt unregistered [ 2665.255040] Key type ._llcrypt registered [ 2665.257628] Key type .llcrypt registered [ 2665.386495] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2666.344762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2666.351392] alg: No test for adler32 (adler32-zlib) [ 2667.364719] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2667.613559] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2669.271171] Key type lgssc registered [ 2670.080662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.381793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2677.400371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.865952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.478231] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2689.005961] LustreError: 96907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.023870] LustreError: 96907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2690.347702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2690.800191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2694.399487] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2699.641934] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2702.164738] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2707.996139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2708.312842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.380611] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2718.491900] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2723.706297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2732.000620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.011474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.028055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.053286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.053469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2734.073257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.479882] LustreError: 99159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2735.699254] Lustre: server umount lustre-OST0000 complete [ 2739.168633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.203184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.213024] Lustre: Skipped 2 previous similar messages [ 2744.289630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.293386] Lustre: Skipped 1 previous similar message [ 2745.187197] LustreError: 99361:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2745.190397] LustreError: 99361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2745.247916] Lustre: server umount lustre-MDT0000 complete [ 2748.517535] LustreError: 96889:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240352 with bad export cookie 7902549747651466086 [ 2748.520764] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.523823] LustreError: 96889:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2748.615319] LustreError: 99563:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2748.621414] LustreError: 99563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2748.833421] Lustre: server umount lustre-MDT0001 complete [ 2755.951331] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2757.744423] Key type lgssc unregistered [ 2757.966650] LNet: 100097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.975559] LNetError: 100097:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.991768] LNet: Removed LNI 192.168.201.105@tcp [ 2758.529097] Key type .llcrypt unregistered [ 2758.533568] Key type ._llcrypt unregistered [ 2773.481111] Key type ._llcrypt registered [ 2773.482871] Key type .llcrypt registered [ 2773.577881] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2774.614425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2774.725965] alg: No test for adler32 (adler32-zlib) [ 2775.833165] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2776.013100] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2777.663154] Key type lgssc registered [ 2779.062889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.507425] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.542369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2803.232286] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2808.352269] LustreError: 101313:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 2834.975839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.898805] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2845.893870] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2853.419375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.751248] LustreError: MGC192.168.201.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2853.756699] LustreError: 102225:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2853.760557] LustreError: 102225:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 2853.767038] LustreError: 102225:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2853.771359] LustreError: 101321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240458 with bad export cookie 7241786964751951055 [ 2853.780950] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.899357] Lustre: server umount lustre-MDT0000 complete [ 2853.905146] LustreError: 102225:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2879.970256] LustreError: 102656:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2880.120914] Lustre: server umount lustre-OST0000 complete [ 2887.403850] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 2895.216802] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2903.114531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2908.093627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2913.444505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2919.091313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2926.588607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2926.637931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.865961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2926.933390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2927.037252] Lustre: lustre-MDT0000: new disk, initializing [ 2927.168430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.208804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2931.610841] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2941.764653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2941.815946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2941.934505] Lustre: 104995:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2941.985679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2941.988384] Lustre: Skipped 1 previous similar message [ 2942.086993] Lustre: lustre-MDT0001: new disk, initializing [ 2942.186540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.251810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2942.260088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2942.393264] Lustre: 105798:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2942.416799] Lustre: 105798:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2942.431911] Lustre: 105798:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2942.446660] Lustre: 105798:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2942.457332] Lustre: 105798:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2942.460290] Lustre: 105798:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2945.845981] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2952.943942] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2955.926924] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2961.883390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2961.954594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2962.158879] Lustre: lustre-OST0000: new disk, initializing [ 2962.164276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2962.256854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.514706] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2969.229422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2969.254961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2969.341923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2974.340679] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2978.423666] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2978.646439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2982.753901] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2982.996095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2984.932325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2984.945063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.980956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2984.990791] Lustre: Skipped 1 previous similar message [ 2990.051076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2990.063339] Lustre: Skipped 1 previous similar message [ 2995.169667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.179823] Lustre: Skipped 1 previous similar message [ 2999.263221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2999.309130] LustreError: 107934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2999.312063] LustreError: 107934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2999.496419] Lustre: server umount lustre-OST0000 complete [ 3003.871988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3003.874298] LustreError: Skipped 1 previous similar message [ 3003.882258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.898938] Lustre: Skipped 1 previous similar message [ 3003.903509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.415543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.176373] LustreError: 108136:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3009.187701] LustreError: 108136:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3009.345577] Lustre: server umount lustre-MDT0000 complete [ 3012.546900] LustreError: 104987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240616 with bad export cookie 7241786964751951153 [ 3012.557557] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3012.661500] LustreError: 108338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3012.668211] LustreError: 108338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3012.876939] Lustre: server umount lustre-MDT0001 complete [ 3020.410229] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:30:24 (1761240624) [ 3023.047770] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3039.415995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3050.112472] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3059.982233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3065.376376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3070.746044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.665855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3083.660630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3083.792651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.093635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3084.115860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3084.231307] Lustre: lustre-MDT0000: new disk, initializing [ 3084.363421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.384979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3088.946496] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3101.046642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.168677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3101.260175] Lustre: 111478:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3101.334935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3101.337895] Lustre: Skipped 1 previous similar message [ 3101.385502] Lustre: lustre-MDT0001: new disk, initializing [ 3101.509772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3101.568773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3101.585210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3105.258229] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3112.618518] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3116.260530] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3122.967110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.076245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.441793] Lustre: lustre-OST0000: new disk, initializing [ 3123.453591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3123.582516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3128.925827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3128.930540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3129.042464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3129.462067] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3136.649636] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3140.475770] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3140.640935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3144.484431] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3144.738563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3149.795625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3149.799476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.802923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.804593] Lustre: Skipped 1 previous similar message [ 3149.820617] Lustre: Skipped 2 previous similar messages [ 3153.087433] LustreError: 114418:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3153.090322] LustreError: 114418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3153.327809] Lustre: server umount lustre-OST0000 complete [ 3157.993857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3158.003237] LustreError: Skipped 1 previous similar message [ 3158.005673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3158.014601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3158.032417] Lustre: Skipped 1 previous similar message [ 3163.532358] LustreError: 114620:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3163.549539] LustreError: 114620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3163.733507] Lustre: server umount lustre-MDT0000 complete [ 3165.157364] LustreError: 111488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.186799] LustreError: 111488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3167.460329] LustreError: 111468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240771 with bad export cookie 7241786964751952070 [ 3167.466672] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.484435] LustreError: 111468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3167.797799] Lustre: server umount lustre-MDT0001 complete [ 3175.541590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.942137] LustreError: 115298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.036031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.990786] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3181.031731] LustreError: 115297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.054123] LustreError: 115297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3186.154928] LustreError: 115297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.500752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3191.967077] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3197.164994] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3200.406978] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3206.981547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3212.172104] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3217.766091] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3228.636961] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3234.278185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.281748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.309920] Lustre: Skipped 1 previous similar message [ 3234.324546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.327235] Lustre: Skipped 2 previous similar messages [ 3237.428147] LustreError: 117642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3237.430437] LustreError: 117642:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3237.658556] Lustre: server umount lustre-OST0000 complete [ 3247.725170] Lustre: server umount lustre-MDT0000 complete [ 3251.689445] LustreError: 116011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.730823] LustreError: 116011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3251.815996] LustreError: 115277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240856 with bad export cookie 7241786964751952994 [ 3251.825275] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.855815] LustreError: 115277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3252.356296] Lustre: server umount lustre-MDT0001 complete [ 3262.069571] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3264.840601] Key type lgssc unregistered [ 3265.048679] LNet: 118579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.061651] LNetError: 118579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3266.094092] LNet: Removed LNI 192.168.201.105@tcp [ 3266.930087] Key type .llcrypt unregistered [ 3266.933103] Key type ._llcrypt unregistered [ 3282.973427] Key type ._llcrypt registered [ 3282.976395] Key type .llcrypt registered [ 3283.062717] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3296.209200] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3297.150903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3297.182404] alg: No test for adler32 (adler32-zlib) [ 3298.161404] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 3298.346309] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3300.015130] Key type lgssc registered [ 3300.845726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.887905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3312.332906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3318.597478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3324.837992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3331.466667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3331.526610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3331.560079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.852647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3332.885541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3332.952911] Lustre: lustre-MDT0000: new disk, initializing [ 3333.034590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.053724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3336.879209] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3346.794635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3346.852487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3346.921452] Lustre: 121702:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3346.984872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3346.987789] Lustre: Skipped 1 previous similar message [ 3347.031333] Lustre: lustre-MDT0001: new disk, initializing [ 3347.126196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3347.170544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3347.182576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3347.261765] Lustre: 122500:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3347.274426] Lustre: 122500:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3347.287339] Lustre: 122500:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3347.294471] Lustre: 122500:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3347.302541] Lustre: 122500:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3347.310473] Lustre: 122500:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3350.528848] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3357.646422] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3360.878630] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3367.181612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3367.265425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3367.439666] Lustre: lustre-OST0000: new disk, initializing [ 3367.447221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3367.538492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.810146] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3375.681870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3375.720909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3375.885944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3381.449822] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3385.885874] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3386.130683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3389.537282] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3389.722876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3391.463405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.470563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3391.489400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.580610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.590387] Lustre: Skipped 2 previous similar messages [ 3397.573420] LustreError: 124645:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3397.774580] Lustre: server umount lustre-OST0000 complete [ 3401.700161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.716181] Lustre: Skipped 1 previous similar message [ 3401.732712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.737882] Lustre: Skipped 1 previous similar message [ 3406.820570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.822692] Lustre: Skipped 1 previous similar message [ 3407.011245] LustreError: 124848:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3407.018949] LustreError: 124848:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3407.111948] Lustre: server umount lustre-MDT0000 complete [ 3410.318663] LustreError: 123450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241014 with bad export cookie 3124326047597234039 [ 3410.322891] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.329076] LustreError: 123450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3410.466526] LustreError: 125050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3410.515318] LustreError: 125050:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3410.737731] Lustre: server umount lustre-MDT0001 complete [ 3418.254877] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:37:02 (1761241022) [ 3424.407932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.719933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.087785] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3429.859380] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.874828] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3434.982562] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.906365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3436.297590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3439.944743] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3444.874676] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3448.714915] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3451.872874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3451.880890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.889426] Lustre: Skipped 1 previous similar message [ 3451.894763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.946825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.610040] LustreError: 127068:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3456.616590] LustreError: 127068:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3456.735931] Lustre: server umount lustre-MDT0000 complete [ 3460.066191] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.082024] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3460.959689] LustreError: 125702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241065 with bad export cookie 3124326047597234963 [ 3460.968719] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.120735] LustreError: 127269:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3461.127207] LustreError: 127269:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3461.364861] Lustre: server umount lustre-MDT0001 complete [ 3468.740698] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:37:53 (1761241073) [ 3473.597956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3489.119591] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3494.241985] LustreError: 127916:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 3519.903297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.882369] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3528.843104] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3530.700351] LustreError: 128580:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3530.708833] LustreError: 128580:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3530.895585] Lustre: server umount lustre-OST0000 complete [ 3537.371395] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:39:01 (1761241141) [ 3542.630294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.940983] LustreError: 129251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.012261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.044108] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3548.128814] LustreError: 129251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.538498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3555.704969] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3560.806091] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3564.544353] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3569.859052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3574.878585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3579.982822] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3598.818829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3598.827715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.851567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.859718] Lustre: Skipped 2 previous similar messages [ 3601.475905] LustreError: 131517:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3601.479159] LustreError: 131517:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3601.589060] Lustre: server umount lustre-MDT0000 complete [ 3604.929817] LustreError: 131518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241209 with bad export cookie 3124326047597235621 [ 3604.948181] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.951261] LustreError: 131518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3604.978301] LustreError: 129247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.244310] Lustre: server umount lustre-MDT0001 complete [ 3614.888584] Lustre: server umount lustre-OST0000 complete [ 3621.762370] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:40:26 (1761241226) [ 3626.455601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.803994] LustreError: 132589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.869519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.872836] Lustre: Skipped 2 previous similar messages [ 3630.168897] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3636.861971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3640.101637] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3644.526950] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3648.150510] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3654.036821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3658.451585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3663.321227] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3667.026575] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3672.762937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3673.724579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3677.413472] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3677.632603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3683.302555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.302590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.306935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.312532] Lustre: Skipped 4 previous similar messages [ 3683.326690] Lustre: Skipped 5 previous similar messages [ 3685.640496] LustreError: 135379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3685.648176] LustreError: 135379:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3685.806394] Lustre: server umount lustre-OST0000 complete [ 3693.549687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3693.561720] LustreError: Skipped 1 previous similar message [ 3695.440619] Lustre: server umount lustre-MDT0000 complete [ 3698.201964] LustreError: 132572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241302 with bad export cookie 3124326047597236699 [ 3698.211178] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3698.557122] Lustre: server umount lustre-MDT0001 complete [ 3705.594065] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:41:50 (1761241310) [ 3711.593833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3727.199604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3732.319478] LustreError: 136425:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 3757.983307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.993498] Lustre: Skipped 2 previous similar messages [ 3762.556125] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3766.689740] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3771.598245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.931361] LustreError: 136451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.948806] LustreError: 136451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3772.081946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3774.877374] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3781.482627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3784.533297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3788.706324] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3791.221197] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3793.932425] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3794.063829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3797.348464] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3797.566179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3802.596800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.600474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3802.600759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.600765] Lustre: Skipped 2 previous similar messages [ 3802.609732] Lustre: Skipped 3 previous similar messages [ 3813.343154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3813.395160] LustreError: 139204:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3813.397488] LustreError: 139204:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3813.452090] Lustre: server umount lustre-MDT0000 complete [ 3815.839551] LustreError: 136433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241420 with bad export cookie 3124326047597237721 [ 3815.844739] LustreError: 136433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3815.851582] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.087891] Lustre: server umount lustre-MDT0001 complete [ 3827.902453] Lustre: server umount lustre-OST0000 complete [ 3833.596592] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:43:58 (1761241438) [ 3837.617511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3853.087542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3858.207480] LustreError: 140255:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 3889.073470] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3893.498430] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3898.753093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.094195] LustreError: 140282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.101159] LustreError: 140282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3899.188260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.191499] Lustre: Skipped 3 previous similar messages [ 3899.274070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3902.077739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3908.603970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3912.061235] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3916.666168] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3919.833432] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3924.941887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3925.015248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3925.123354] Lustre: lustre-OST0001: new disk, initializing [ 3925.131230] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3929.571211] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3934.780193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3934.793686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3934.881370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3935.981852] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3939.547535] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3939.769358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3943.087365] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3943.298632] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3944.932285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3944.949754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.959294] Lustre: Skipped 1 previous similar message [ 3944.963176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3944.965827] Lustre: Skipped 8 previous similar messages [ 3959.263247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3959.304457] LustreError: 143987:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3959.315941] LustreError: 143987:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3959.365048] Lustre: server umount lustre-OST0000 complete [ 3968.922355] Lustre: server umount lustre-OST0001 complete [ 3979.229700] Lustre: server umount lustre-MDT0000 complete [ 3982.492352] LustreError: 144392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241586 with bad export cookie 3124326047597238666 [ 3982.495495] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.548314] LustreError: 144392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3983.005212] Lustre: server umount lustre-MDT0001 complete [ 3988.322213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.165494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3995.277747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.294353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4005.732236] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:46:49 (1761241609) [ 4007.544880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4009.413086] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4011.073552] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:46:55 (1761241615) [ 4017.754710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.974145] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4018.013598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4021.977593] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4030.668097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4030.796444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4030.866814] Lustre: 146232:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4034.746847] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4038.839859] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4041.747449] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4047.017627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.158822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4052.032124] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4054.641284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4056.834638] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4060.194656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4060.375329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4063.900539] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4064.116834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4069.857720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.867853] LustreError: Skipped 4 previous similar messages [ 4069.871648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.883102] Lustre: Skipped 5 previous similar messages [ 4071.822697] Lustre: server umount lustre-OST0000 complete [ 4091.196096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4095.529646] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4095.539899] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4095.719624] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4100.201450] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4104.672954] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4109.861571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4109.874297] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4110.178287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4113.389992] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4113.641470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4117.064391] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c4c47174000.ost_server_uuid 50 [ 4118.600786] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c4c47174000.ost_server_uuid in FULL state after 0 sec [ 4118.779554] LustreError: 146239:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.5@tcp) returned error from blocking AST (req@00000000ee78cafa x1846794940303104 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000091192956/0x2b5bd5fca588a2dd lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.5@tcp remote: 0x5d0368439970f5d7 expref: 5 pid: 147884 timeout: 4218 lvb_type: 0 [ 4118.797171] LustreError: lustre-MDT0000: A client on nid 192.168.201.5@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4118.803034] LustreError: 146228:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.5@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000091192956/0x2b5bd5fca588a2dd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.5@tcp remote: 0x5d0368439970f5d7 expref: 6 pid: 147884 timeout: 0 lvb_type: 0 [ 4125.153657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4125.156665] Lustre: Skipped 15 previous similar messages [ 4128.292368] Lustre: server umount lustre-OST0000 complete [ 4133.858092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4133.871577] LustreError: Skipped 2 previous similar messages [ 4137.880513] Lustre: server umount lustre-MDT0000 complete [ 4140.513633] LustreError: 147884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4140.534084] LustreError: 147884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4141.046938] LustreError: 146975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241745 with bad export cookie 3124326047597239842 [ 4141.052801] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.359148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4155.681126] Lustre: server umount lustre-MDT0001 complete [ 4163.148758] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4165.551756] Key type lgssc unregistered [ 4165.765814] LNet: 151763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4165.770042] LNetError: 151763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.779581] LNet: Removed LNI 192.168.201.105@tcp [ 4166.385276] Key type .llcrypt unregistered [ 4166.388829] Key type ._llcrypt unregistered [ 4185.829644] Key type ._llcrypt registered [ 4185.832394] Key type .llcrypt registered [ 4185.941684] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4186.912225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4186.975640] alg: No test for adler32 (adler32-zlib) [ 4188.110679] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 4188.279751] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4189.935177] Key type lgssc registered [ 4191.133787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4198.687664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4198.705227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.066914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.706411] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4209.636513] LustreError: 153134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.672544] LustreError: 153134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4211.296038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4211.625213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.024184] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4219.591883] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4222.841584] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4228.283295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4228.611966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.578206] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4237.950143] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4245.055815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4253.946334] Lustre: Failing over lustre-MDT0000 [ 4254.129383] LustreError: 155403:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4254.225682] Lustre: server umount lustre-MDT0000 complete [ 4254.928794] LustreError: 153131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.205862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4255.223590] Lustre: Skipped 2 previous similar messages [ 4260.321858] LustreError: 153135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.347573] LustreError: 153135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4265.445989] LustreError: 153135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.462015] LustreError: 153135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4266.689282] LustreError: 153115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.5@tcp arrived at 1761241871 with bad export cookie 17108356828745761387 [ 4270.561593] LustreError: 153135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.578367] LustreError: 153135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4271.583129] Lustre: 152448:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761241859/real 1761241859] req@00000000540c9315 x1846795873314432/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761241875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4271.598379] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4272.539758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.808976] LustreError: 153134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.825112] LustreError: 153134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4281.031287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.071943] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4284.254347] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4286.310689] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4286.317749] Lustre: lustre-MDT0000: Denying connection for new client 37ac155f-0289-4dd3-8468-db933d2a9df2 (at 192.168.201.5@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4286.449680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4296.683024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.686794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4296.697729] Lustre: Skipped 1 previous similar message [ 4300.927299] LustreError: 156338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4300.930117] LustreError: 156338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4301.143384] Lustre: server umount lustre-OST0000 complete [ 4304.473868] LustreError: 156540:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4304.479523] Lustre: 155906:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4304.483651] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4304.496691] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4304.510396] LustreError: 155906:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b8341907 x1846795873334528/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4304.525549] LustreError: 155906:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4304.530623] LustreError: 155906:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4304.536550] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 0 recovered and 2 were evicted. [ 4304.702698] LustreError: 156540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4304.708304] LustreError: 156540:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4304.815150] Lustre: server umount lustre-MDT0000 complete [ 4306.912288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.923443] LustreError: 153129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.265184] LustreError: 153116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241912 with bad export cookie 17108356828745761716 [ 4308.265948] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.274014] LustreError: 153116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4308.301314] LustreError: 152444:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4310.532039] LustreError: 156742:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4310.546706] LustreError: 156742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4310.873776] Lustre: server umount lustre-MDT0001 complete [ 4321.530272] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4324.062799] Key type lgssc unregistered [ 4324.338642] LNet: 157276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4324.349259] LNetError: 157276:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4325.420905] LNet: Removed LNI 192.168.201.105@tcp [ 4326.099487] Key type .llcrypt unregistered [ 4326.102160] Key type ._llcrypt unregistered [ 4346.357590] Key type ._llcrypt registered [ 4346.359097] Key type .llcrypt registered [ 4346.435531] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4347.507523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4347.525637] alg: No test for adler32 (adler32-zlib) [ 4348.590204] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 4348.894655] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4350.559169] Key type lgssc registered [ 4351.821587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.438484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4358.458428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.867485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.408945] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4370.408588] LustreError: 158651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.429604] LustreError: 158651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4371.587367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4372.091600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4376.431715] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4382.386795] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4386.823667] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4394.972454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4395.375884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.039409] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4405.623988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4405.709684] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4410.869097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4410.892978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4412.931150] LustreError: 160904:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4413.107897] Lustre: server umount lustre-OST0000 complete [ 4418.025333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4418.046071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.055693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4421.091418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4421.094848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.209495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4426.211688] Lustre: Skipped 2 previous similar messages [ 4431.328249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4431.329420] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 4431.335366] Lustre: Skipped 1 previous similar message [ 4431.407314] LustreError: 161104:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4431.425579] LustreError: 161104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4431.508617] Lustre: server umount lustre-MDT0000 complete [ 4434.158232] LustreError: 158631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242038 with bad export cookie 17877566457392048486 [ 4434.159721] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.167254] LustreError: 158631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4434.271083] LustreError: 161308:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4434.274994] LustreError: 161308:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4434.495501] Lustre: server umount lustre-MDT0001 complete [ 4442.397238] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4444.764236] Key type lgssc unregistered [ 4444.999308] LNet: 161839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.009388] LNetError: 161839:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4446.056579] LNet: Removed LNI 192.168.201.105@tcp [ 4446.763529] Key type .llcrypt unregistered [ 4446.766416] Key type ._llcrypt unregistered [ 4472.112366] Key type ._llcrypt registered [ 4472.114363] Key type .llcrypt registered [ 4472.247212] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4472.998532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4473.076844] alg: No test for adler32 (adler32-zlib) [ 4474.177329] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 4474.424909] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4476.111163] Key type lgssc registered [ 4477.230994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4484.338858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4484.352094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.862856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.547427] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4496.360916] LustreError: 163502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4496.393258] LustreError: 163502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4500.962701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4501.578554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4506.605216] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4512.754805] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4516.359540] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4523.169858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4523.530368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.696616] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4530.677792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4533.088634] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4540.895995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4540.899435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.901379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4540.913412] Lustre: Skipped 1 previous similar message [ 4543.684287] LustreError: 165750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4543.884001] Lustre: server umount lustre-OST0000 complete [ 4547.557262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4547.560050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.576772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.581985] Lustre: Skipped 1 previous similar message [ 4551.140740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4551.141655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.395384] LustreError: 165952:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4553.408298] LustreError: 165952:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4553.491637] Lustre: server umount lustre-MDT0000 complete [ 4556.256288] LustreError: 164721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.267311] LustreError: 164721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4556.375470] LustreError: 163482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242160 with bad export cookie 3438062390721299736 [ 4556.378459] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4556.381196] LustreError: 163482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4556.529248] LustreError: 166154:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4556.542465] LustreError: 166154:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4556.833312] Lustre: server umount lustre-MDT0001 complete [ 4565.445714] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4567.777976] Key type lgssc unregistered [ 4568.028961] LNet: 166687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4568.033428] LNetError: 166687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4568.062183] LNet: Removed LNI 192.168.201.105@tcp [ 4568.772516] Key type .llcrypt unregistered [ 4568.774896] Key type ._llcrypt unregistered [ 4584.848440] Key type ._llcrypt registered [ 4584.850247] Key type .llcrypt registered [ 4584.937375] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4585.576855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4585.590933] alg: No test for adler32 (adler32-zlib) [ 4586.550271] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 4586.721302] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4588.351198] Key type lgssc registered [ 4589.312940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4597.413352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4597.431727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.704594] Lustre: *** cfs_fail_loc=135, val=0*** [ 4598.791489] LustreError: 168171:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4598.796770] LustreError: 168171:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -2 [ 4598.801774] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4598.801774] [ 4598.806678] LustreError: MGC192.168.201.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4598.814965] LustreError: 168131:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4598.820243] LustreError: 168131:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 4598.825676] LustreError: 168131:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4598.831864] LustreError: 168131:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4598.936806] Lustre: server umount lustre-MDT0000 complete [ 4598.938830] LustreError: 168131:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4603.889976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.311871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.401719] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4614.420185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4614.702209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4617.844904] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4621.969299] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4624.613284] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4625.868529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.876774] Lustre: Skipped 1 previous similar message [ 4625.876957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.884328] Lustre: Skipped 1 previous similar message [ 4631.008585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.015436] Lustre: Skipped 1 previous similar message [ 4636.129716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.140579] Lustre: Skipped 1 previous similar message [ 4640.223646] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4640.229207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4640.236808] Lustre: Skipped 1 previous similar message [ 4640.302681] LustreError: 169887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4640.320267] LustreError: 169887:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4640.406862] Lustre: server umount lustre-MDT0000 complete [ 4643.878516] LustreError: 168523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242248 with bad export cookie 13852128050771454261 [ 4643.882877] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4643.888468] LustreError: 168523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4644.018654] LustreError: 170091:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4644.030943] LustreError: 170091:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4644.236771] Lustre: server umount lustre-MDT0001 complete [ 4654.251266] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4657.189667] Key type lgssc unregistered [ 4657.572671] LNet: 170625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4657.583810] LNetError: 170625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4657.608223] LNet: Removed LNI 192.168.201.105@tcp [ 4658.510862] Key type .llcrypt unregistered [ 4658.514444] Key type ._llcrypt unregistered [ 4683.768981] Key type ._llcrypt registered [ 4683.771160] Key type .llcrypt registered [ 4683.983354] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4684.362505] Key type .llcrypt unregistered [ 4684.365464] Key type ._llcrypt unregistered [ 4700.052992] Key type ._llcrypt registered [ 4700.054552] Key type .llcrypt registered [ 4700.144943] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4700.837757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4700.869717] alg: No test for adler32 (adler32-zlib) [ 4701.906476] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 4702.060574] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4703.711163] Key type lgssc registered [ 4705.090341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4711.281757] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4711.300468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4726.879472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4731.999308] LustreError: 172719:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 4758.624281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.832746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4768.492709] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4773.314859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.670592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.717566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4776.360362] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4778.978758] LustreError: 172745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.986083] LustreError: 172745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4782.267987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4782.542456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4785.190651] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4788.917095] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4791.315235] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4793.046489] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4803.046856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4803.050297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.051665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4803.056083] Lustre: Skipped 1 previous similar message [ 4808.161966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.164843] Lustre: Skipped 2 previous similar messages [ 4813.283894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4813.290142] Lustre: Skipped 1 previous similar message [ 4814.303122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4814.342461] LustreError: 175299:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4814.495489] Lustre: server umount lustre-OST0000 complete [ 4818.401724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.407064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4818.417323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4818.423674] Lustre: Skipped 1 previous similar message [ 4823.509438] LustreError: 175501:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4823.514353] LustreError: 175501:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4823.520580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4823.524053] Lustre: Skipped 1 previous similar message [ 4823.575973] Lustre: server umount lustre-MDT0000 complete [ 4826.343260] LustreError: 172728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242430 with bad export cookie 3426651500318868712 [ 4826.349367] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4826.356507] LustreError: 172728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4826.470919] LustreError: 175703:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4826.474525] LustreError: 175703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4826.628340] Lustre: server umount lustre-MDT0001 complete [ 4834.658689] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4836.839807] Key type lgssc unregistered [ 4837.145764] LNet: 176234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4837.158130] LNetError: 176234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4837.168963] LNet: Removed LNI 192.168.201.105@tcp [ 4837.865308] Key type .llcrypt unregistered [ 4837.871832] Key type ._llcrypt unregistered [ 4855.332654] Key type ._llcrypt registered [ 4855.334383] Key type .llcrypt registered [ 4855.423187] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4856.709483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4856.757561] alg: No test for adler32 (adler32-zlib) [ 4857.727318] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 4857.934139] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4859.591223] Key type lgssc registered [ 4860.559800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4867.366963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4867.385964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.806983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.705538] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4879.329567] LustreError: 177617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.360928] LustreError: 177617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4879.373158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4879.806434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.896655] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4888.914294] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4892.514738] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4899.103992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4899.465861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4904.472595] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4910.276573] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4913.841348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4919.320615] Lustre: Failing over lustre-MDT0000 [ 4919.448130] LustreError: 179920:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4919.563313] Lustre: server umount lustre-MDT0000 complete [ 4920.800295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4920.805541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.820165] LustreError: 178636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.048882] LustreError: 177610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4924.393874] LustreError: 178636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4924.398814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.413437] Lustre: Skipped 1 previous similar message [ 4929.507923] LustreError: 177617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.531052] LustreError: 177617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4936.486913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.632964] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.823041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.861647] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4937.401238] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4939.863894] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4942.312157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4942.320322] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4942.346788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4943.212657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4947.808650] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4957.666045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4957.671165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.681914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4962.273362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4962.283672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.317803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4963.711199] LustreError: 181156:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4963.720163] LustreError: 181156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4963.825688] Lustre: server umount lustre-OST0000 complete [ 4967.908290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4967.909471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.924651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.940664] Lustre: Skipped 1 previous similar message [ 4967.965410] Lustre: Skipped 1 previous similar message [ 4972.708850] LustreError: 181357:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4972.717506] LustreError: 181357:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4972.823217] Lustre: server umount lustre-MDT0000 complete [ 4973.025864] LustreError: 177617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.035208] LustreError: 177617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4975.764262] LustreError: 177596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242580 with bad export cookie 14973885083547660623 [ 4975.766437] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4975.780387] LustreError: 177596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4975.923083] LustreError: 181560:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4975.934167] LustreError: 181560:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4976.123209] Lustre: server umount lustre-MDT0001 complete [ 4984.535893] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4986.205456] Key type lgssc unregistered [ 4986.421792] LNet: 182092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4986.427630] LNetError: 182092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4987.493360] LNet: Removed LNI 192.168.201.105@tcp [ 4987.952222] Key type .llcrypt unregistered [ 4987.953947] Key type ._llcrypt unregistered [ 5002.785041] Key type ._llcrypt registered [ 5002.786705] Key type .llcrypt registered [ 5002.870300] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5003.599480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5003.675530] alg: No test for adler32 (adler32-zlib) [ 5004.718375] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5004.936617] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5006.607187] Key type lgssc registered [ 5007.608362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5013.355820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5013.370314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.653554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.203373] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5023.224058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5023.449264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.837714] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5029.607039] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5032.434502] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5038.328378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5038.689708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5039.735232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5043.120186] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5047.438979] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5050.650980] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5058.941446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5070.505466] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5081.057294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5081.060860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.080331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.157813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.163566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.164150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.168889] Lustre: Skipped 1 previous similar message [ 5086.465077] LustreError: 185912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5086.574364] Lustre: server umount lustre-OST0000 complete [ 5090.274500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.277467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5090.292391] Lustre: Skipped 1 previous similar message [ 5090.292765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5090.296102] Lustre: Skipped 1 previous similar message [ 5095.394029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5095.398755] Lustre: Skipped 1 previous similar message [ 5100.519976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.524541] Lustre: Skipped 1 previous similar message [ 5103.071152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5103.163134] LustreError: 186113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5103.165701] LustreError: 186113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5103.241917] Lustre: server umount lustre-MDT0000 complete [ 5105.638526] LustreError: 185090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5105.674787] LustreError: 185090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5106.435880] LustreError: 183446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242710 with bad export cookie 5991541627802764089 [ 5106.441353] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5106.450781] LustreError: 183446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5106.590881] LustreError: 186316:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5106.593439] LustreError: 186316:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5106.780814] Lustre: server umount lustre-MDT0001 complete [ 5115.233722] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5117.668642] Key type lgssc unregistered [ 5117.961708] LNet: 186847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5117.975121] LNetError: 186847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5117.994098] LNet: Removed LNI 192.168.201.105@tcp [ 5118.768430] Key type .llcrypt unregistered [ 5118.770940] Key type ._llcrypt unregistered [ 5135.801519] Key type ._llcrypt registered [ 5135.803261] Key type .llcrypt registered [ 5135.864864] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5136.405281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5136.414756] alg: No test for adler32 (adler32-zlib) [ 5137.431760] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5137.615343] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5139.255190] Key type lgssc registered [ 5140.099702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5145.841319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5145.852410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.231844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5150.572471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5157.346925] LustreError: 188220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.374342] LustreError: 188220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5158.379987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5158.650347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.843821] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5166.063411] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5168.745452] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5174.299422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5174.757371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5179.449598] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5183.190250] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5192.186257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5198.051891] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5209.126900] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5217.043850] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5224.528812] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5233.123876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5233.129521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.133540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.146089] Lustre: Skipped 1 previous similar message [ 5233.151537] Lustre: Skipped 1 previous similar message [ 5238.246095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5238.250279] Lustre: Skipped 1 previous similar message [ 5239.038943] LustreError: 192074:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5239.199527] Lustre: server umount lustre-OST0000 complete [ 5243.364086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.366629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5243.375017] Lustre: Skipped 1 previous similar message [ 5243.389845] Lustre: Skipped 1 previous similar message [ 5248.136183] LustreError: 192276:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5248.140746] LustreError: 192276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5248.239616] Lustre: server umount lustre-MDT0000 complete [ 5248.498902] LustreError: 188219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.505670] LustreError: 188219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5250.913491] LustreError: 188199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242855 with bad export cookie 4265179226118246420 [ 5250.917836] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.919529] LustreError: 188199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5251.015754] LustreError: 192477:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5251.019150] LustreError: 192477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5251.176638] Lustre: server umount lustre-MDT0001 complete [ 5258.430641] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5260.389180] Key type lgssc unregistered [ 5260.609573] LNet: 193009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5260.614414] LNetError: 193009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5260.625858] LNet: Removed LNI 192.168.201.105@tcp [ 5261.186277] Key type .llcrypt unregistered [ 5261.188105] Key type ._llcrypt unregistered [ 5281.699461] Key type ._llcrypt registered [ 5281.701056] Key type .llcrypt registered [ 5281.886681] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5282.861621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5282.875127] alg: No test for adler32 (adler32-zlib) [ 5284.159340] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5284.622585] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5286.375395] Key type lgssc registered [ 5287.533545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5296.585477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5296.633143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.296419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.597264] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5307.873812] LustreError: 194404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.887515] LustreError: 194404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5310.385306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5310.722453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5313.604144] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5318.069317] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5320.775768] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5325.454843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5325.718126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.706603] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5333.635883] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5343.157685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5356.512655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5356.521394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.535712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.565654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.569097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.570528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.585472] Lustre: Skipped 1 previous similar message [ 5362.050258] LustreError: 196828:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5362.252608] Lustre: server umount lustre-OST0000 complete [ 5367.267175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5367.270638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.284092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5372.383972] LustreError: 197030:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5372.386976] LustreError: 197030:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5372.446215] Lustre: server umount lustre-MDT0000 complete [ 5373.931651] LustreError: 195479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.952514] LustreError: 195479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5375.205925] LustreError: 196030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242979 with bad export cookie 1415028377130166180 [ 5375.207662] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.219981] LustreError: 196030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5375.321592] LustreError: 197232:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5375.324113] LustreError: 197232:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5375.470436] Lustre: server umount lustre-MDT0001 complete [ 5382.672747] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5384.794840] Key type lgssc unregistered [ 5385.076914] LNet: 197764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5385.087138] LNetError: 197764:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.157743] LNet: Removed LNI 192.168.201.105@tcp [ 5386.771481] Key type .llcrypt unregistered [ 5386.776099] Key type ._llcrypt unregistered [ 5399.190646] Key type ._llcrypt registered [ 5399.193504] Key type .llcrypt registered [ 5399.252839] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5399.806584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5399.821289] alg: No test for adler32 (adler32-zlib) [ 5400.786655] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5400.941533] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5402.567190] Key type lgssc registered [ 5403.287101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.884679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5408.898058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.321552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.283032] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5419.716632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5420.020990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5422.340071] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5425.872893] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5428.349352] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5433.495717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5433.822629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.893232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5438.498907] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5443.222751] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5455.334896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.341607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.343318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5457.462180] LustreError: 201387:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5457.584385] Lustre: server umount lustre-OST0000 complete [ 5460.453943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.463981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.468980] Lustre: Skipped 2 previous similar messages [ 5460.480410] Lustre: Skipped 2 previous similar messages [ 5465.573106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5465.577831] Lustre: Skipped 1 previous similar message [ 5466.385861] LustreError: 201589:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5466.393773] LustreError: 201589:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5466.496912] Lustre: server umount lustre-MDT0000 complete [ 5469.161102] LustreError: 201388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243073 with bad export cookie 8943055781945876838 [ 5469.163906] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.174572] LustreError: 201388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5469.282957] LustreError: 201791:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5469.289659] LustreError: 201791:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5469.491881] Lustre: server umount lustre-MDT0001 complete [ 5476.970734] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5479.963559] Key type lgssc unregistered [ 5480.325935] LNet: 202323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5480.345420] LNetError: 202323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5480.375340] LNet: Removed LNI 192.168.201.105@tcp [ 5481.310222] Key type .llcrypt unregistered [ 5481.313176] Key type ._llcrypt unregistered [ 5496.826371] Key type ._llcrypt registered [ 5496.828472] Key type .llcrypt registered [ 5496.943244] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5497.864469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5497.899327] alg: No test for adler32 (adler32-zlib) [ 5498.886509] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5499.039515] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5500.671439] Key type lgssc registered [ 5501.462637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5506.974766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5506.992577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.420328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.702611] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5517.796842] LustreError: 203548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.807448] LustreError: 203548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5518.187130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5518.522704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5521.391945] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5525.459973] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5527.821683] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5532.277412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5532.578677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5533.617600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5536.843727] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5540.916627] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5548.811220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5548.866610] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5549.099433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5553.377034] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5557.520740] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5558.910805] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5570.167930] Lustre: Permanently deactivating lustre-OST0001 [ 5570.182876] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5577.274232] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5583.962219] Lustre: Permanently reactivating lustre-OST0001 [ 5583.984745] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5583.987751] Lustre: Skipped 2 previous similar messages [ 5586.427423] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.444036] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5586.453558] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5586.465065] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5586.500837] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5596.642899] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5596.646546] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.652536] LustreError: Skipped 1 previous similar message [ 5596.655516] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5596.665210] Lustre: Skipped 2 previous similar messages [ 5596.679469] Lustre: Skipped 1 previous similar message [ 5599.729684] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5599.750773] Lustre: Skipped 1 previous similar message [ 5602.650542] LustreError: 207008:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5602.861081] Lustre: server umount lustre-OST0001 complete [ 5604.841748] LustreError: 205145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.867958] LustreError: 205145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5609.440550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5609.446944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.487480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5609.962654] LustreError: 205145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.989530] LustreError: 205145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5614.145639] LustreError: 207210:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5614.154219] LustreError: 207210:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5614.303906] Lustre: server umount lustre-OST0000 complete [ 5620.193066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.198607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5620.205056] Lustre: Skipped 2 previous similar messages [ 5620.212508] Lustre: Skipped 2 previous similar messages [ 5623.619632] LustreError: 207411:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5623.623149] LustreError: 207411:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5623.698316] Lustre: server umount lustre-MDT0000 complete [ 5625.315084] LustreError: 204261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.336351] LustreError: 204261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5627.227833] LustreError: 203529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243231 with bad export cookie 8333212474507306614 [ 5627.238321] LustreError: 203529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.239253] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.387327] LustreError: 207616:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5627.390925] LustreError: 207616:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5627.738988] Lustre: server umount lustre-MDT0001 complete [ 5636.634717] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5639.279679] Key type lgssc unregistered [ 5639.473756] LNet: 208147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5639.488913] LNetError: 208147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5639.501947] LNet: Removed LNI 192.168.201.105@tcp [ 5640.223322] Key type .llcrypt unregistered [ 5640.224748] Key type ._llcrypt unregistered [ 5665.655810] Key type ._llcrypt registered [ 5665.657920] Key type .llcrypt registered [ 5665.716696] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5674.588645] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5675.308485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5675.335553] alg: No test for adler32 (adler32-zlib) [ 5676.299367] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5676.446097] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5678.079177] Key type lgssc registered [ 5678.907779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5685.672116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5691.986747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5697.840351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5704.751070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5712.274884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5712.320729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5712.328763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.617726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5713.658741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5713.742417] Lustre: lustre-MDT0000: new disk, initializing [ 5713.853518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.885659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5717.345368] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5725.177556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5725.229486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5725.295292] Lustre: 212073:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5725.346614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5725.351238] Lustre: Skipped 1 previous similar message [ 5725.421236] Lustre: lustre-MDT0001: new disk, initializing [ 5725.510876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.552099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5725.562670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5725.637483] Lustre: 212868:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5725.650063] Lustre: 212868:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5725.661189] Lustre: 212868:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5725.676791] Lustre: 212868:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5725.680801] Lustre: 212868:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5725.694119] Lustre: 212868:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5729.330424] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5734.795210] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5737.194478] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5742.236639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5742.282900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5742.450176] Lustre: lustre-OST0000: new disk, initializing [ 5742.453608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5742.541052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.853118] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5750.847974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5750.859890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5750.977271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5752.674445] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5755.918264] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5756.105375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5758.905632] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5759.152313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5761.509985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5761.514394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.523426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5761.528627] Lustre: Skipped 1 previous similar message [ 5766.457098] LustreError: 215010:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5766.582132] Lustre: server umount lustre-OST0000 complete [ 5771.750537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5771.756954] LustreError: Skipped 1 previous similar message [ 5771.761602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.764348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5771.777628] Lustre: Skipped 2 previous similar messages [ 5776.865086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.876364] Lustre: Skipped 2 previous similar messages [ 5780.968084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.971036] Lustre: Skipped 1 previous similar message [ 5784.031274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5784.179059] LustreError: 215212:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5784.185054] LustreError: 215212:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5784.356306] Lustre: server umount lustre-MDT0000 complete [ 5788.719755] LustreError: 212063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243393 with bad export cookie 4806975824299796902 [ 5788.723120] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.734616] LustreError: 212063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5788.894569] LustreError: 215415:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5788.904281] LustreError: 215415:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5789.186778] Lustre: server umount lustre-MDT0001 complete [ 5795.932312] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 14:16:40 (1761243400) [ 5801.816555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.133739] LustreError: 216086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.145293] LustreError: 216086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5802.192443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.501074] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5807.585592] LustreError: 216086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.682279] LustreError: 216086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.749755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5812.135412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.266789] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5819.177689] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5821.699033] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5826.490790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5826.746206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.331655] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5835.923374] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5838.061220] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5847.696856] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5851.906800] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5863.007797] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5876.701388] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5876.707160] Lustre: Skipped 1 previous similar message [ 5894.454820] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5894.459657] Lustre: Skipped 1 previous similar message [ 5928.815612] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5928.820456] Lustre: Skipped 4 previous similar messages [ 6002.551458] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6002.555885] Lustre: Skipped 8 previous similar messages [ 6011.365768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6011.369928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6011.399728] Lustre: Skipped 1 previous similar message [ 6011.400420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6011.408038] Lustre: Skipped 1 previous similar message [ 6014.017206] LustreError: 219591:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6014.027588] LustreError: 219591:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6014.174888] Lustre: server umount lustre-OST0000 complete [ 6021.600720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.618024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6021.626252] Lustre: Skipped 1 previous similar message [ 6023.626553] LustreError: 219798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6023.639829] LustreError: 219798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6023.765213] Lustre: server umount lustre-MDT0000 complete [ 6026.721853] LustreError: 216104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.742031] LustreError: 216104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6027.556962] LustreError: 216805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243631 with bad export cookie 4806975824299797826 [ 6027.559554] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6027.562476] LustreError: 216805:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6027.942807] Lustre: server umount lustre-MDT0001 complete [ 6035.809820] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6038.242430] Key type lgssc unregistered [ 6038.508859] LNet: 220531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6038.514437] LNetError: 220531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6038.531217] LNet: Removed LNI 192.168.201.105@tcp [ 6039.324790] Key type .llcrypt unregistered [ 6039.327056] Key type ._llcrypt unregistered [ 6055.449633] Key type ._llcrypt registered [ 6055.451209] Key type .llcrypt registered [ 6055.518192] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6056.129236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6056.139685] alg: No test for adler32 (adler32-zlib) [ 6057.233552] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 6057.415105] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6059.063193] Key type lgssc registered [ 6059.960321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6064.921182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6064.937620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.350128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.165269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6075.885044] LustreError: 221903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.896697] LustreError: 221903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6076.079417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6076.415442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6080.059946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6084.710113] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6087.916225] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6093.970681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6094.424601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.333268] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6104.229231] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6131.689178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6131.690507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.691718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6131.712835] Lustre: Skipped 1 previous similar message [ 6132.344938] LustreError: 224332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6132.461581] Lustre: server umount lustre-OST0000 complete [ 6136.803034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.805759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6136.820206] Lustre: Skipped 1 previous similar message [ 6136.860265] Lustre: Skipped 1 previous similar message [ 6141.924537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.927783] Lustre: Skipped 1 previous similar message [ 6142.082518] LustreError: 224536:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6142.085660] LustreError: 224536:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6142.332692] Lustre: server umount lustre-MDT0000 complete [ 6146.088928] LustreError: 221882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243750 with bad export cookie 3104552946511075209 [ 6146.097470] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.300630] LustreError: 224738:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6146.322318] LustreError: 224738:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6146.636904] Lustre: server umount lustre-MDT0001 complete [ 6156.843710] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6159.442653] Key type lgssc unregistered [ 6159.713690] LNet: 225269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6159.725158] LNetError: 225269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6159.735931] LNet: Removed LNI 192.168.201.105@tcp [ 6160.390691] Key type .llcrypt unregistered [ 6160.393252] Key type ._llcrypt unregistered [ 6176.849310] Key type ._llcrypt registered [ 6176.851112] Key type .llcrypt registered [ 6176.942278] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6177.165140] Key type .llcrypt unregistered [ 6177.166719] Key type ._llcrypt unregistered [ 6197.332954] Key type ._llcrypt registered [ 6197.334576] Key type .llcrypt registered [ 6197.430811] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6208.950605] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6209.766485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6209.802476] alg: No test for adler32 (adler32-zlib) [ 6210.776200] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 6210.996919] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6212.671162] Key type lgssc registered [ 6213.453771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6220.961737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6226.311366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6231.374381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.461138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.329262] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5973 sec ========== 14:24:03 (1761243843) [ 6241.325605] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:24:05 (1761243845) === [ 6244.437721] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:24:08 (1761243848) === [ 6276.653440] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6278.898252] Key type lgssc unregistered [ 6279.152917] LNet: 231358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6279.160482] LNetError: 231358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6279.178169] LNet: Removed LNI 192.168.201.105@tcp [ 6279.870445] Key type .llcrypt unregistered [ 6279.871961] Key type ._llcrypt unregistered